Crashlytics Generate Symbols gradle step fails: GC

Question

I'm converting a project from gradle version 3.3 to 4.10.1. This project is mostly C++ code built using a custom build step - not CMake (externalNativeBuild) or Android.mk(ndkBuild). It produces the following libraries:

armeabi-v7a unstripped: 883.2MB
arm64-v8a unstripped: 864.6MB
armeabi-v7a stripped: 15.6MB
arm64-v8a stripped: 23.9MB

The build process fails at the crashlyticsGenerateSymbolsGoogleDistribution step, with the error OutOfMemoryError: GC Overhead Limit Exceeded.

Is there a method or argument for providing more memory to the crashlytics plugin step?

---

The top level gradle.properties file contains the following jvm args: org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=512m - though I have tried as much as org.gradle.jvmargs=-Xmx12g:MaxPermSize=2g. No changes to these values seem to affect the crashlyticsGenerateSymbols step.

If I run an assembleGoogleDistribution build, skipping the crashlytics step, the build completes with no issues.

Build stack trace:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':application:crashlyticsGenerateSymbolsGoogleDistribution'.
> GC overhead limit exceeded

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':application:crashlyticsGenerateSymbolsGoogleDistribution'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
    at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
    at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
    at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
    at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.interleaveRanges(DwarfDataParser.java:321)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processCompilationUnit(DwarfDataParser.java:194)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:173)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:139)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.parse(DwarfDataParser.java:62)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parseElf(ElfDataParser.java:131)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parse(ElfDataParser.java:101)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parse(ElfDataParser.java:39)
    at com.crashlytics.tools.android.project.codemapping.csym.ElfCSymFactory.createCSymFromFile(ElfCSymFactory.java:99)
    at com.crashlytics.tools.android.project.codemapping.csym.NdkCSymGenerator.generateCodeMappings(NdkCSymGenerator.java:70)
    at com.crashlytics.tools.android.project.CSymManager.generate(CSymManager.java:96)
    at com.crashlytics.tools.android.DeveloperTools.processProperties(DeveloperTools.java:692)
    at com.crashlytics.tools.android.DeveloperTools.processArgsInternal(DeveloperTools.java:348)
    at com.crashlytics.tools.android.DeveloperTools.gradleMain(DeveloperTools.java:292)

---

Gradle and Java info:

./gradlew --version

------------------------------------------------------------
Gradle 4.10.1
------------------------------------------------------------

Build time:   2018-09-12 11:33:27 UTC
Revision:     76c9179ea9bddc32810f9125ad97c3315c544919

Kotlin DSL:   1.0-rc-6
Kotlin:       1.2.61
Groovy:       2.4.15
Ant:          Apache Ant(TM) version 1.9.11 compiled on March 23 2018
JVM:          1.8.0_121 (Oracle Corporation 25.121-b13)
OS:           Mac OS X 10.13.6 x86_64

top level build.gradle

...
buildscript {
    dependencies {
        classpath 'com.android.tools.build:gradle:3.3.0'
        classpath 'com.google.gms:google-services:4.1.0'
        classpath 'io.fabric.tools:gradle:1.31.0'
    }
}
...

application build.gradle

apply plugin: 'com.android.application'
apply plugin: 'io.fabric'
...
crashlytics {
    enableNdk true
    androidNdkOut './../../../build/crashlytics'
    androidNdkLibsOut './../../../build/crashlytics/lib'
}
...
dependencies {
    // Crashlytics Kit
    implementation('com.crashlytics.sdk.android:crashlytics:2.10.1@aar') {
        transitive = true
    }
    // NDK Kit
    implementation('com.crashlytics.sdk.android:crashlytics-ndk:2.1.0@aar') {
        transitive = true
    }
}
...

(I have confirmed that the androidNdkOut path contains armeabi-v7a and arm64-v8a folders which contain our unstripped libnative.so files, and androidNdkLibsOut contains the same folders with our stripped libnative.so files)

---

How can I ensure that the crashlytics plugin is getting enough memory for its crashlyticsGenerateSymbols step so that it doesn't exceed the garbage collector overhead?

Answer 1

I run into this issue and found two walk around. 1: use NDK r18 and Android Gradle plugin 3.4.2. (19 and later NDK will cause java heap issue when generate symbol for arm64-v8a) 2: use most recent NDK r21d and set linker flag -fuse-ld=lld If use ndkBuild, set APP_LDFLAGS += -fuse-ld=lld in application.mk

Those two worked for me to get pass this issue.

Answer 2

Try putting a file fabric.properties in the app directory with the following content

disableEnhancedSymbolGeneration=true

It disables DWARF information processing but retains symbols information in mapping. You will not get file names and lines in resulting stacktraces in this case. Still you will get full function names (including namespaces) instead of function addresses.

Answer 3

Please try to downgrade the Fabric Plugin from 1.31.0 to 1.25.4 in build.gradle

classpath 'io.fabric.tools:gradle:1.25.4'