Android 4.3 On-screen GPU profiling - long gfx wai

I have just updated a Galaxy Nexus to 4.3 and enabled the new On-screen GPU profiling function, and see the following result for Android setup screen:

According to the platform highlights:

[With] colors indicating time spent creating drawing commands (blue), issuing the commands (orange), and waiting for the commands to complete (yellow).

Even on a very simple screen, there are many instances that the screen refresh time is above the threshold for a smooth 60 fps (green line), and it's mostly because there are many instances where a refresh would spend a significant time waiting for the commands to complete (yellow line*), while other times this step is almost instantaneous. This is not something particular to the Setting app either, but seem to be present for all the apps I've tested so far. *looks more orange than yellow to me

What I wish to know are:

Is this time spent "waiting for the commands to complete" means that the screen commands are being actively process and thus the time would accurately represent the time spent drawing the screen. OR does this time include the time waiting for video-sync (though I'd think triple buffer would be used to remove this requirement)?
The time spent "waiting for the commands to complete" would fluctuate wildly even when drawing the same screen (scroll slightly up & down on the same ScrollView), is there any guidance on how to reduce this fluctuation (or if it could be reduced at all)?

[Edit:]

Updated Nexus 7 as well, and it's even worse:

As many as 5 frames are being skipped "waiting for the commands to complete" and it really showed in usage, the app was very choppy and unresponsive.

[Edit 2:] I have performed these per this article to trigger TRIM for ~3 days, so the N7 should be as "pristine" as it's going to get short of a factory reset.

The device has been idle for over an hour
No idle maintenance window event has been performed in the last 24 hours
The device is either being charged with 30 percent battery or it has 80 percent battery

Now Google Maps seem to behave a bit better (see below), so some of the issues may be related to flash access speed though I don't know how.

Still, since the Galaxy Nexus is factory reset, its long "waiting for the commands to complete" time can't be related to the lack of TRIM command, and following the above steps indeed didn't produce improvements. So we are back at square one...

"Waiting for commands to complete" indicates that there are dependencies on rendered frames. For example, the app might be using glReadPixels to read from the rendered frame. This means that after the frame has been sent to the GPU for rendering, the app is blocked until rendering that frame finishes (whereas normally it would be able to continue right away). Android tries to let the app queue up as many rendering commands as possible, so suddenly introducing a wait might actually mean the app has to wait for several previously queued frames to be drawn before the frame it's waiting for is rendered.

glReadPixels isn't the only command that causes this kind of dependency. If an app wants to write to a texture that's currently being used, it has to wait until all the frames that depend on the texture have finished. This is plausibly what is happening with Google Maps: if each map tile is a texture, it might be reusing an old off-screen tile by writing a new tile into it ready to show. Once the app has queued a frame that doesn't use the old tile, it tries to write into that texture, but really the texture is still being used to render previously queued frames. The app has to wait until those frames are finished (and the GPU is no longer reading from the 'unused' texture) before it can write.

In theory, it's possible to have a worker thread write to the texture, allowing the main thread to go on queueing new frames smoothly. But GL's complex thread model makes it very tricky to get something like this right, and the main thread would eventually have to wait for the texture upload to complete anyway.

As for the Settings app, it could be that Android's GL backend is doing the same texture-reuse trick for the icons, but that's just a guess. Perhaps the Galaxy Nexus is using a 2D compositor to do frame composition, which saves power but at the cost of introducing a wait in the driver. I don't know whether that kind of dependency would be measured in the chart.