How compose multiple thumbnails of GLES scenes int

Question

Our app generates a number of scenes using GLES 2. Making a picker (scrolling list of images) to select which scene to go to. The scenes are not available as pre-made images; using GL rendering at the time they are needed. On iOS, we created the scenes in an offscreen pbuffer, as they became visible while the list scrolled.

Attempt #1: On Android, the code equivalent to our iOS code yields GL Errors, because there is no current EGLContext (or it is not a valid time or thread to render to current context). On iOS we found examples of offscreen rendering that did not depend on any association with a GL view. On Android, most examples seem to assume you are inside the drawing code of a GLSurfaceView, so you are already on the GL thread with correct EGLContext. Or they were pre-Lollipop examples, when the framework's internal EGLContext apparently was exposed to the app, so code "just worked", even if it wasn't threadsafe (so eventually causes problems).

Attempt #2: GLSurfaceView per thumbnail. Works great for one thumbnail. Add second thumbnail, emulator crashes with graphics driver error. I think due to two GL threads managed by the GL views.

Attempt #3: Approach used by Grafika as described by fadden: multiple surfaceviews (not GLSurfaceViews), manage own GL thread to render to those surfaceviews. This would likely work if not inside a scrolling list, could change to textureviews (but must share GL thread or maybe EGLContext's in a shared group - see Grafika) to work better with scrolling, but still not a good idea for more than 2 to 4 thumbnails, as per fadden's comment about performance cliff when hit the compositor's limit on N number of surfaces.

Attempt #4: GL with no surfaceviews at all. Manage own GL thread (a la Grafika). When thumbnail draws, render GL into an FBO or pbuffer. Convert to bitmap. Display that bitmap in the thumbnail view. Note that this is different than examples I've seen, because the bitmap doesn't exist at the time the view is created; it only exists when the view goes to draw. Cache the bitmaps so can reuse on each draw.

For #4, here are the subtasks:

1. An EGLContext and GL thread.
2. Create FBO of correct size, or use our pbuffer code that already works on iOS. (Probably was copied from an ancient example; I understand that FBO would be more efficient solution).
3. Render GL into buffer. [This code already works; app-specific.]
4. Capture buffer into an Android bitmap. [I have an example of this; not shown here.]
5. Dynamically display bitmap in view. For example, attach bitmap to an ImageView.

(1.) is the subtask I am asking how to do.

---

NOTE #1: Could avoid all this by making the entire scrolling view a single GL surface, simulating list scrolling, rendering each scene into an appropriate rectangle - but that would complicate our cross-platform code (Xamarin), and eliminate flexibility. Seeking a solution that fits into Android's view hierarchy, so can add text captions, or any other desired change. Then we would have a general-purpose ability to render GL anywhere we want, and compose that with non-GL features provided by Android.

NOTE #2: Unlike Grafika, don't need the GL rendering to change every frame. That is why we don't need multiple surfaceviews - once the bitmaps exist, we can reuse them. So the solution won't be multiple real-time GL views. If you are seeking that, limit the number of views, and use Grafika's approach with several SurfaceViews (or TextureViews if need to move them around).

Answer 1

NOTE: This is Xamarin C# code. That is why it has different capitalization and other naming differences (java get/set methods replaced by C# properties). It uses Xamarin wrappers for Android java, a java version would be a 1:1 translation of each line of code.

OurGLRenderer is a custom class to manage an EGLContext. This allows GL rendering without a GLSurfaceView or TextureView.

The heart of this class is "MakeCurrent": after calling that, you can make GL calls, because you have an active EGLContext. The GL calls render to an offscreen buffer, previously created in CreateGLAndSurface via CreateOffscreenBuffer.

To instead render to a TextureView (or SurfaceView?), then use CreateWindowSurface instead of CreateOffscreenBuffer.

using System;

using Android.Graphics;
using Android.Runtime;

using Javax.Microedition.Khronos.Egl;

namespace YourAppNameHere
{
    // Manage an EGLContext. This allows GL rendering without a GLSurfaceView or TextureView.
    // The heart of this class is "MakeCurrent": after calling that, you can make GL calls,
    //   because you have an active EGLContext.
    // The GL calls render to an offscreen buffer, previously created in CreateGLAndSurface via CreateOffscreenBuffer.
    // To instead render to a `TextureView` (or `SurfaceView`?), then use `CreateWindowSurface` instead of `CreateOffscreenBuffer`.
    public class OurGLRenderer
    {
        // Your app supplies this class.
        public interface IRenderEngine
        {
            void EnsureInitialized();
            // The frame buffer or view size.
            void EnsureSize( int width, int height );
            // Our client calls our MakeCurrent, then calls this to render.
            // "model" should be a class in your app.
            // On Android, this could return a Bitmap, which you then place in an ImageView.
            object RenderAsPlatformImage( object model );
        }

        // HACK: ASSUMES Singleton.
        public static Action OneTimeAfterCreated;
        // Most recent error code.
        static int _error = 0;


        #region "=== static methods - could be in a utility class ==="
        // These are static, so that they can be used independently. Could be "public".
        // ----- Based on https://forums.xamarin.com/discussion/3406/xamarin-android-textureview-sample-render-an-opengl-scene-to-a-view -----
        static bool InitializeEGL( out IEGL10 _egl10, out EGLDisplay _display, out bool _display_initialized,
                                   out bool _choose_config, out EGLConfig _config )
        {
            _display_initialized = false;
            _choose_config = false;
            _config = null;

            //FAIL Javax.Microedition.Khronos.Egl.IEGL10 t_egl10 = (Javax.Microedition.Khronos.Egl.IEGL10)Javax.Microedition.Khronos.Egl.EGLContext.EGL;
            _egl10 = EGLContext.EGL.JavaCast<IEGL10>();

            // _display
            _display = _egl10.EglGetDisplay( EGL10.EglDefaultDisplay ); // EglGetCurrentDisplay returns NULL !
            if (_display == null)
                return false;

            // EglInitialize
            int[] _major_minor = new int[ 2 ];
            _display_initialized = _egl10.EglInitialize( _display, _major_minor );
            Console.WriteLine( string.Format( "EglInitialize -> {0}, version={1}.{2}", _display_initialized, _major_minor[ 0 ], _major_minor[ 1 ] ) );
            if (!CheckEglError( _egl10, "EglInitialize" ) || !_display_initialized)
                return false;

            return InitializeEGLConfig( _egl10, _display, out _choose_config, out _config );
        }

        static bool InitializeEGLConfig( IEGL10 _egl10, EGLDisplay _display,
                                         out bool _choose_config, out EGLConfig _config )
        {
            _config = null;


            // EglChooseConfig -> OpenGL ES 2.0 Config
            int EGL_OPENGL_ES2_BIT = 4;
            int[] _attribs_config = new int[]{
                        EGL10.EglRenderableType, EGL_OPENGL_ES2_BIT, // IMPORTANT
                        EGL10.EglRedSize, 8,
                        EGL10.EglGreenSize, 8,
                        EGL10.EglBlueSize, 8,
                        EGL10.EglAlphaSize, 8,
                        EGL10.EglDepthSize, 0,
                        EGL10.EglStencilSize, 0,
                        EGL10.EglNone
                    };
            EGLConfig[] _configs = null;
            _configs = new EGLConfig[ 1 ];
            int[] _numconfigs = new int[ 1 ];
            _choose_config = _egl10.EglChooseConfig( _display, _attribs_config, _configs, 1, _numconfigs );
            if (!CheckEglError( _egl10, "EglChooseConfig" ) || !_choose_config)
                return false;

            _config = _configs[ 0 ];
            // Why?  (I guess so not holding another reference.)
            _configs[ 0 ] = null; _configs = null;

            return (_config != null);
        }

        static bool EglCreateContext( IEGL10 _egl10, EGLDisplay _display, EGLConfig _config,
                                      out EGLContext _context )
        {
            // EglCreateContext -> OpenGL ES 2.0 Context
            int EGL_CONTEXT_CLIENT_VERSION = 0x3098;
            int _version = EGL10.EglVersion;
            int[] _attribs_config = new int[]{
                        EGL_CONTEXT_CLIENT_VERSION, 2, // IMPORTANT
                        EGL10.EglNone
                    };
            _context = _egl10.EglCreateContext( _display, _config, EGL10.EglNoContext, _attribs_config );
            return CheckEglError( _egl10, "EglCreateContext" ) && (_context != null);
        }

        static bool CreateWindowSurface( IEGL10 _egl10, EGLDisplay _display, EGLConfig _config, SurfaceTexture _surfaceTexture,
                                         out EGLSurface _surface )
        {
            _surface = null;


            if (_surfaceTexture == null)
                return false;


            // EglCreateWindowSurface
            int[] _attribs_config = new int[]{
                        EGL10.EglNone
                    };
            _surface = _egl10.EglCreateWindowSurface( _display, _config, _surfaceTexture, _attribs_config );
            return CheckEglError( _egl10, "EglCreateWindowSurface" ) && (_surface != null);
        }

        static bool CreateOffscreenBuffer( IEGL10 _egl10, EGLDisplay _display, EGLConfig _config, int width, int height,
                                           out EGLSurface _surface )
        {
            int[] _attribs_config = new int[]{
                        EGL10.EglWidth, width,
                        EGL10.EglHeight, height,
                        EGL10.EglNone
                    };
            _surface = _egl10.EglCreatePbufferSurface( _display, _config, _attribs_config );
            return CheckEglError( _egl10, "EglCreatePbufferSurface" ) && (_surface != null);
        }

        static bool EglMakeCurrent( IEGL10 _egl10, EGLDisplay _display, EGLSurface _surface, EGLContext _context,
                                    out bool _make_current )
        {
            _make_current = _egl10.EglMakeCurrent( _display, _surface, _surface, _context );
            return (CheckEglError( _egl10, "EglMakeCurrent" ) && _make_current);
        }

        static bool EglSwapBuffers( IEGL10 _egl10, EGLDisplay _display, EGLSurface _surface )
        {
            bool _ok = _egl10.EglSwapBuffers( _display, _surface );
            return (CheckEglError( _egl10, "EglSwapBuffers" ) && _ok);
        }

        static bool CheckEglError( IEGL10 _egl10, string tag )
        {
            _error = _egl10.EglGetError();
            if (_error != EGL10.EglSuccess) {
                Log.e( tag, string.Format( "EGL-Error={0}", _error ) );
                return false;
            }
            return true;
        }
        #endregion


        #region "=== constructor and Dispose ==="
        public OurGLRenderer( int ourWidth, int ourHeight )
        {
            Width = ourWidth; Height = ourHeight;
            EnsureOurRenderEngine( ourWidth, ourHeight );
        }

        public void Dispose()
        {
            OurRenderEngine = null;
            EndOurGL();
        }
        #endregion


        #region "=== public fields ==="
        public int Width { get; protected set; }
        public int Height { get; protected set; }

        public IRenderEngine OurRenderEngine;
        #endregion


        #region "=== public methods ==="
        // NOTE: call EndOurGL when leave fragment.
        public bool BeginOurGL( int width, int height )
        {
            if (alreadyBeginningOurGL)
                // CAUTION: Caller must not call EndOurGLThread - might be another view starting it!
                return false;

            alreadyBeginningOurGL = true;
            try {
                if (!EnsureGLAndSurfaceInitialized( width, height )) {
                    return false;
                }

                //TEST_TextureView( _egl10, null );   // tmstest

                return true;

            } finally {
                alreadyBeginningOurGL = false;
            }
        }

        // Client must call this before any GL calls.
        // Before first GL call, and whenever Android may have done drawing in its own EGLContext.
        public bool MakeCurrent()
        {
            return EglMakeCurrent( _egl10, _display, _surface, _context, out _make_current );
        }

        // ASSUME MakeCurrent already called.
        public void EnsureOurSize( int ourWidth, int ourHeight )
        {
            // In our app, we create an OurGLRenderer, then use it to render multiple images of the same size -
            // our IRenderEngine is set up once for that size.
            // You might not have this constraint; in which case, comment this out.
            if (Width != ourWidth || Height != ourHeight)
                throw new InvalidProgramException( "OurGLRenderer.EnsureOurSize - all images must be same size." );

            OurRenderEngine.EnsureSize( Width, Height );
        }

        public void EndOurGL()
        {
            EndAndDispose( _egl10 ); _egl10 = null;
        }
        #endregion


        #region "=== private fields ==="
        IEGL10 _egl10 = null;
        EGLDisplay _display = null;
        bool _display_initialized = false;
        bool _choose_config = false;
        EGLConfig _config = null;
        EGLSurface _surface = null;
        EGLContext _context = null;
        bool _make_current = false;

        bool alreadyBeginningOurGL = false;
        #endregion


        #region "=== private methods ==="
        bool EnsureGLAndSurfaceInitialized( int width, int height )
        {
            if (_surface == null) {
                if (!CreateGLAndSurface( width, height ))
                    return false;
            }

            // TODO: Try this, once NOT on PaintingView.
            if (false) {
                // Make current. We aren't rendering yet, but confirm that this succeeds.
                if (!MakeCurrent()) {
                    // Failed; undo any work that was done.
                    EndOurGL();
                    return false;
                }
            }


            return true;
        }

        bool CreateGLAndSurface( int width, int height )
        {
            if (!InitializeEGL( out _egl10, out _display, out _display_initialized, out _choose_config, out _config ) ||
                !EglCreateContext( _egl10, _display, _config, out _context ) ||
                !CreateOffscreenBuffer( _egl10, _display, _config, width, height, out _surface )) {
                // Failed; undo any work that was done.
                EndOurGL();
                return false;
            }

            return true;
        }


        void EndAndDispose( IEGL10 _egl10 )
        {
            // EglMakeCurrent
            if (_make_current) {
                _egl10.EglMakeCurrent( _display, EGL10.EglNoSurface, EGL10.EglNoSurface, EGL10.EglNoContext );
                _make_current = false;
            }
            // EglDestroyContext
            if (_context != null) {
                _egl10.EglDestroyContext( _display, _context );
                _context = null;
            }
            // EglDestroySurface
            if (_surface != null) {
                _egl10.EglDestroySurface( _display, _surface );
                _surface = null;
            }
            //
            if (_config != null) {
                _config.Dispose();
                _config = null;
            }
            // EglTerminate
            if (_display_initialized) {
                _egl10.EglTerminate( _display );
                _display_initialized = false;
            }
            //
            if (_display != null) {
                _display.Dispose();
                _display = null;
            }
        }
        #endregion


        #region "=== specific to our app ==="
        // Shows that OurRenderEngine must be created, and BeginOurGL called.
        // Also shows a call to MakeCurrent, and OurRenderEngine.EnsureInitialized.
        public void EnsureOurRenderEngine( int ourWidth, int ourHeight )
        {
        //  if ((OurRenderEngine == null) || !ReferenceEquals( AppState.ActiveRenderEngine, OurRenderEngine )) {
        //      AppState.ReleaseRenderEngine();
        //      // NOTE: We can't pass a reDrawDelegate because multiple views are sharing this engine.
        //      //AppState.ActiveRenderEngine = OurRenderEngine = AppState.CreateRenderEngine( MainActivity.OurAppType, ourWidth, ourHeight, null );
        //      AppState.ActiveRenderEngine = OurRenderEngine = (OffscreenRenderEngine)AppState.CreateRenderEngine( AppState.AppType.Offscreen, ourWidth, ourHeight, null );
        //
        //      if (Width != ourWidth || Height != ourHeight)
        //          throw new InvalidProgramException( "OurGLRenderer.EnsureOurRenderEngine - all images must be same size." );
        //
        //      // BEFORE ActiveRenderEngine.EnsureInitialized.
        //      // TODO: GL Program fails to compile, when called in OnDraw. Conflict with framework's GL context?
        //      BeginOurGL( ourWidth, ourHeight );
        //
        //      // Before any GL calls.
        //      if (!MakeCurrent())
        //          return;
        //      // Needed because FragmentMain.OnCreateView runs before this, so its initialization is skipped (no ActiveRenderEngine yet)..
        //      OurRenderEngine.EnsureInitialized();
        //
        //      if (OneTimeAfterCreated != null) {
        //          OneTimeAfterCreated();
        //          OneTimeAfterCreated = null;
        //      }
        //  }
        }
        #endregion
    }
}