How to handle :java.util.concurrent.TimeoutExcepti

Question

We're seeing a number of TimeoutExceptions in GcWatcher.finalize, BinderProxy.finalize, and PlainSocketImpl.finalize. 90+% of them happen on Android 4.3. We're getting reports of this from Crittercism from users out in the field.

The error is a variation of: "com.android.internal.BinderInternal$GcWatcher.finalize() timed out after 10 seconds"

java.util.concurrent.TimeoutException: android.os.BinderProxy.finalize() timed out after 10 seconds
at android.os.BinderProxy.destroy(Native Method)
at android.os.BinderProxy.finalize(Binder.java:459)
at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:187)
at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:170)
at java.lang.Thread.run(Thread.java:841)

So far we haven't had any luck reproducing the problem in house or figuring out what might have caused it.

Any ideas what can cause this? Any idea how to debug this and find out which part of the app causes this? Anything that sheds light on the issue helps.

More Stacktraces:

1   android.os.BinderProxy.destroy  
2   android.os.BinderProxy.finalize Binder.java, line 482
3   java.lang.Daemons$FinalizerDaemon.doFinalize    Daemons.java, line 187
4   java.lang.Daemons$FinalizerDaemon.run   Daemons.java, line 170
5   java.lang.Thread.run    Thread.java, line 841  

2

1   java.lang.Object.wait   
2   java.lang.Object.wait   Object.java, line 401
3   java.lang.ref.ReferenceQueue.remove ReferenceQueue.java, line 102
4   java.lang.ref.ReferenceQueue.remove ReferenceQueue.java, line 73
5   java.lang.Daemons$FinalizerDaemon.run   Daemons.java, line 170
6   java.lang.Thread.run

3

1   java.util.HashMap.newKeyIterator    HashMap.java, line 907
2   java.util.HashMap$KeySet.iterator   HashMap.java, line 913
3   java.util.HashSet.iterator  HashSet.java, line 161
4   java.util.concurrent.ThreadPoolExecutor.interruptIdleWorkers    ThreadPoolExecutor.java, line 755
5   java.util.concurrent.ThreadPoolExecutor.interruptIdleWorkers    ThreadPoolExecutor.java, line 778
6   java.util.concurrent.ThreadPoolExecutor.shutdown    ThreadPoolExecutor.java, line 1357
7   java.util.concurrent.ThreadPoolExecutor.finalize    ThreadPoolExecutor.java, line 1443
8   java.lang.Daemons$FinalizerDaemon.doFinalize    Daemons.java, line 187
9   java.lang.Daemons$FinalizerDaemon.run   Daemons.java, line 170
10  java.lang.Thread.run

4

1   com.android.internal.os.BinderInternal$GcWatcher.finalize   BinderInternal.java, line 47
2   java.lang.Daemons$FinalizerDaemon.doFinalize    Daemons.java, line 187
3   java.lang.Daemons$FinalizerDaemon.run   Daemons.java, line 170
4   java.lang.Thread.run

Answer 1

I found some slides about this issue.

http://de.slideshare.net/DroidConTLV/android-crash-analysis-and-the-dalvik-garbage-collector-tools-and-tips

In this slides the author tells that it seems to be a problem with GC, if there are a lot of objects or huge objects in heap. The slide also include a reference to a sample app and a python script to analyze this issue.

https://github.com/oba2cat3/GCTest

https://github.com/oba2cat3/logcat2memorygraph

Furthermore I found a hint in comment #3 on this side: https://code.google.com/p/android/issues/detail?id=53418#c3

Answer 2

One thing which is invariably true is that at this time, the device would be suffocating for some memory (which is usually the reason for GC to most likely get triggered).

As mentioned by almost all authors earlier, this issue surfaces when Android tries to run GC while the app is in background. In most of the cases where we observed it, user paused the app by locking their screen. This might also indicate memory leak somewhere in the application, or the device being too loaded already. So the only legitimate way to minimize it is:

• to ensure there are no memory leaks, and
• to reduce the memory footprint of the app in general.

Answer 3

Full disclosure - I'm the author of the previously mentioned talk in TLV DroidCon.

I had a chance to examine this issue across many Android applications, and discuss it with other developers who encountered it - and we all got to the same point: this issue cannot be avoided, only minimized.

I took a closer look at the default implementation of the Android Garbage collector code, to understand better why this exception is Thrown and on what could be the possible causes. I even found a possible root cause during experimentation.

The root of the problem is at the point a device "Goes to Sleep" for a while - this means that the OS has decided to lower the battery consumption by stopping most User Land processes for a while, and turning Screen off, reducing CPU cycles, etc. The way this is done - is on a Linux system level where the processes are Paused mid run. This can happen at any time during normal Application execution, but it will stop at at a Native system call, as the context switching is done on the kernel level. So - this is where the Dalvik GC joins the story. The Dalvik GC code (as implemented in the Dalvik project in the AOSP site) is not a complicated piece of code. The basic way it work is covered in my DroidCon slides. what I did not cover is the basic GC loop - at the point where the collector has a list of Objects to finalize (and destroy). the loop logic at the base can be simplified like this:

1. take starting_timestamp,
2. remove object for list of objects to release,
3. release object - finalize() and call native destroy() if required,
4. take end_timestamp,
5. calculate (end_timestamp - starting_timestamp) and compare against a hard coded timeout value of 10 seconds,
6. if timeout has reached - throw the concurrent.TimeoutException and kill the process.

Now consider the following scenario:

Application runs along doing its thing. this is not a User facing application, it runs in the background. During this background operation, Objects are created, used and need to be collected to release memory. Application does not bother with a Wakelock - as this will affect the battery adversely, and seems unnecessary. this means the Application will invoke the GC from time to time. Normally the GC runs is completed without a hitch. Sometimes (very rarely) the System will decide to Sleep in the middle of the GC run. This will happen if you run your application long enough, and monitor the Dalvik memory logs closely. Now - consider the timestamp logic of the basic GC loop - it is possible for the device to start the run, take a start_stamp, and go to sleep at the destroy() native call on a system object. when it wakes up and resumes the run, the destroy() will finish, and the next end_stamp will be the time it took the destroy() call+the sleep time. If the sleep time was long - over 10 seconds, the concurrent.timeout exception will be thrown.

I have seen this in the graphs generated from the analysis python script - for Android System Applications, not just my own monitored apps. collect enough logs, you will eventually see it.

Bottom line:

The issue cannot be avoided - you will encounter it if your app runs in the background. You can mitigate by taking a wakelock, and prevent the device from sleeping, but that is a different story altogether, and a new headache, and maybe another talk in another con.

You can minimize the problem by reducing GC calls - making the scenario less likely. tips are in the slides.

I have not yet had the chance to go over the Dalvik 2 (a.k.a ART) GC code - which boasts a new Generational Compacting feature, or performed any experiments on an Android Lollipop.

Added 7/5/2015:

After reviewing the Crash reports aggregation for this crash type, it looks like these crashes from version 5.0+ of Android OS (Lollipop with ART) only account for .5% of this crash type. This means that the ART GC changes has reduced the frequency of these crashes.

Added 6/1/2016:

Looks like the Android project has added a lot of info on how the GC works in Dalvik 2.0 (a.k.a ART). You can read about it here - Debugging ART Garbage Collection . It also discusses some tools to get information on the GC behavior for your app. Sending a SIGQUIT to your app process will essentially cause an ANR, and dump the application state to a log file for analysis.

Answer 4

We solved the problem by stopping the FinalizerWatchdogDaemon.

public static void fix() {
    try {
        Class clazz = Class.forName("java.lang.Daemons$FinalizerWatchdogDaemon");

        Method method = clazz.getSuperclass().getDeclaredMethod("stop");
        method.setAccessible(true);

        Field field = clazz.getDeclaredField("INSTANCE");
        field.setAccessible(true);

        method.invoke(field.get(null));

    }
    catch (Throwable e) {
        e.printStackTrace();
    }
}

You can call the method in Application's lifecycle, like attachBaseContext(). For the same reason, you also can specific the phone's manufacture to fix the problem, it's up to you.

Answer 5

The finalizeQueue may be too long

i think that java may require GC.SuppressFinalize() & GC.ReRegisterForFinalize() to let user reduce the finalizedQueue length explicitly

if the JVM' source code is available, may implement these method ourself, such as android ROM maker

Answer 6

We see this constantly, all over our app, using Crashlytics. The crash usually happens way down in platform code. A small sampling:

android.database.CursorWindow.finalize() timed out after 10 seconds

java.util.regex.Matcher.finalize() timed out after 10 seconds

android.graphics.Bitmap$BitmapFinalizer.finalize() timed out after 10 seconds

org.apache.http.impl.conn.SingleClientConnManager.finalize() timed out after 10 seconds

java.util.concurrent.ThreadPoolExecutor.finalize() timed out after 10 seconds

android.os.BinderProxy.finalize() timed out after 10 seconds

android.graphics.Path.finalize() timed out after 10 seconds

The devices on which this happens are overwhelmingly (but not exclusively) devices manufactured by Samsung. That could just mean that most of our users are using Samsung devices; alternately it could indicate a problem with Samsung devices. I'm not really sure.

I suppose this doesn't really answer your questions, but I just wanted to reinforce that this seems quite common, and is not specific to your application.