MediaCodec.configure fails with IllegalStateExcept

Question

I've been using grafika's MoviePlayer and bigFlake's ExtractMpegFramesTest to implement a seeking and extract frame feature in our app. These work fine for most of our user, however some of them encounter a IllegalStateException on MediaCodec.configure when setting up the ExtractMpegFramesTest (this happens on Samsung Galaxy S4 mini, Samsung Galaxy J7, Samsung Galaxy A5, Huawei Ascend G7). The relevant code is as the following:

MoviePlayer

public void prepareResources() throws IOException {
    // The MediaExtractor error messages aren't very useful.  Check to see if the input
    // file exists so we can throw a better one if it's not there.
    if (!mSourceFile.canRead()) {
        throw new FileNotFoundException("Unable to read " + mSourceFile);
    }

    try {
        mExtractor = new MediaExtractor();
        mExtractor.setDataSource(mSourceFile.toString());
        mTrackIndex = selectTrack(mExtractor);
        if (mTrackIndex < 0) {
            throw new RuntimeException("No video track found in " + mSourceFile);
        }
        mExtractor.selectTrack(mTrackIndex);

        MediaFormat format = mExtractor.getTrackFormat(mTrackIndex);

        // Create a MediaCodec decoder, and configure it with the MediaFormat from the
        // extractor.  It's very important to use the format from the extractor because
        // it contains a copy of the CSD-0/CSD-1 codec-specific data chunks.
        String mime = format.getString(MediaFormat.KEY_MIME);
        mDecoder = MediaCodec.createDecoderByType(mime);
        mDecoder.configure(format, mOutputSurface, null, 0);
        mDecoder.start();
        mDecoderInputBuffers = mDecoder.getInputBuffers();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

ExtractMpegFramesTest

private void extractMpegFrames() throws IOException {
    MediaCodec decoder = null;
    CodecOutputSurface outputSurface = null;
    MediaExtractor extractor = null;
    /*int saveWidth = 640;
    int saveHeight = 480;*/

    try {
        long start1 = System.currentTimeMillis();

        File inputFile = new File(mInputVideoPath);   // must be an absolute path
        // The MediaExtractor error messages aren't very useful.  Check to see if the input
        // file exists so we can throw a better one if it's not there.
        if (!inputFile.canRead()) {
            throw new FileNotFoundException("Unable to read " + inputFile);
        }

        extractor = new MediaExtractor();
        extractor.setDataSource(inputFile.toString());
        int trackIndex = selectTrack(extractor);
        if (trackIndex < 0) {
            throw new RuntimeException("No video track found in " + inputFile);
        }
        extractor.selectTrack(trackIndex);

        MediaFormat format = extractor.getTrackFormat(trackIndex);
        if (VERBOSE) {
            Log.d(TAG, "Video size is " + format.getInteger(MediaFormat.KEY_WIDTH) + "x" +
                    format.getInteger(MediaFormat.KEY_HEIGHT));
        }

        // Could use width/height from the MediaFormat to get full-size frames.
        outputSurface = new CodecOutputSurface(format.getInteger(MediaFormat.KEY_WIDTH), format.getInteger(MediaFormat.KEY_HEIGHT));

        // Create a MediaCodec decoder, and configure it with the MediaFormat from the
        // extractor.  It's very important to use the format from the extractor because
        // it contains a copy of the CSD-0/CSD-1 codec-specific data chunks.
        String mime = format.getString(MediaFormat.KEY_MIME);
        decoder = MediaCodec.createDecoderByType(mime);
        decoder.configure(format, outputSurface.getSurface(), null, 0); //fails right here
        decoder.start();

        long end1 = System.currentTimeMillis();
        Timber.d("extractMpegFrames(): FRAME_EXTRACT setup in: %d millis", (end1-start1));

        long start2 = System.currentTimeMillis();
        doExtract(extractor, trackIndex, decoder, outputSurface);
        long end2 = System.currentTimeMillis();
        Timber.d("extractMpegFrames(): FRAME_EXTRACT doExtract in: %d millis", (end2-start2));
    } finally {
        // release everything we grabbed
        if (outputSurface != null) {
            outputSurface.release();
            outputSurface = null;
        }
        if (decoder != null) {
            decoder.stop();
            decoder.release();
            decoder = null;
        }
        if (extractor != null) {
            extractor.release();
            extractor = null;
        }
    }
}

I'm aware of this question, however what I don't understand is why MediaCodec can be configured OK in MoviePlayer (the video runs just fine) but it fails in ExtractMpegFramesTest. At first, I think the device has some issues with the OpenGL set up, but it turn out to be MediaCodec problem.

Any insight would be greatly appreciated.

EDIT: After obtaining a Galaxy S4 mini testing device, I've been able to get a much more helpful log:

D/MoviePlayer: Extractor selected track 0 (video/avc): {max-input-size=1572864, height=1080, csd-0=java.nio.ByteArrayBuffer[position=0,limit=20,capacity=20], width=1920, durationUs=5131211, csd-1=java.nio.ByteArrayBuffer[position=0,limit=9,capacity=9], mime=video/avc, isDMCMMExtractor=1}
D/MoviePlayer: Video size is 1920x1080
D/MoviePlayer: Extractor selected track 0 (video/avc): {max-input-size=1572864, height=1080, csd-0=java.nio.ByteArrayBuffer[position=0,limit=20,capacity=20], width=1920, durationUs=5131211, csd-1=java.nio.ByteArrayBuffer[position=0,limit=9,capacity=9], mime=video/avc, isDMCMMExtractor=1}
I/OMXClient: Using client-side OMX mux.
E/ACodec: [OMX.qcom.video.decoder.avc] storeMetaDataInBuffers failed w/ err -2147483648
E/ACodec:  configureCodec multi window instance fail  appPid : 3283
E/ACodec: [OMX.qcom.video.decoder.avc] configureCodec returning error -38
E/MediaCodec: Codec reported an error. (omx error 0x80001001, internalError -38)
W/System.err: java.lang.IllegalStateException
W/System.err:     at android.media.MediaCodec.native_configure(Native Method)
W/System.err:     at android.media.MediaCodec.configure(MediaCodec.java:262)
W/System.err:     at co.test.testing.player.MoviePlayer.prepareResources(MoviePlayer.java:237)
W/System.err:     at co.test.testing.activities.VideoActivity.surfaceCreated(VideoActivity.java:276)
W/System.err:     at android.view.SurfaceView.updateWindow(SurfaceView.java:602)
W/System.err:     at android.view.SurfaceView.access$000(SurfaceView.java:94)
W/System.err:     at android.view.SurfaceView$3.onPreDraw(SurfaceView.java:183)
W/System.err:     at android.view.ViewTreeObserver.dispatchOnPreDraw(ViewTreeObserver.java:891)
W/System.err:     at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2201)
W/System.err:     at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1256)
W/System.err:     at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:6635)
W/System.err:     at android.view.Choreographer$CallbackRecord.run(Choreographer.java:813)
W/System.err:     at android.view.Choreographer.doCallbacks(Choreographer.java:613)
W/System.err:     at android.view.Choreographer.doFrame(Choreographer.java:583)
W/System.err:     at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:799)
W/System.err:     at android.os.Handler.handleCallback(Handler.java:733)
W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:95)
W/System.err:     at android.os.Looper.loop(Looper.java:146)
W/System.err:     at android.app.ActivityThread.main(ActivityThread.java:5593)
W/System.err:     at java.lang.reflect.Method.invokeNative(Native Method)
W/System.err:     at java.lang.reflect.Method.invoke(Method.java:515)
W/System.err:     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1283)
W/System.err:     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1099)
W/System.err:     at dalvik.system.NativeStart.main(Native Method)

These are the codec info retrieved from devices having this problem:

• OMX.qcom.video.encoder.avc

• OMX.SEC.avc.enc

• OMX.Exynos.AVC.Encoder

The strange thing is the Galaxy S4 mini can handle 720p videos just fine, it's just having problem with 1080p videos.

Answer 1

Ok, here to answer my own question again:

• Firstly, why I can't extract frame with ExtractMpegFramesTest after using MoviePlayer to display videos:

  It appears that some devices can't handle 2 instances of MediaCodec at the same time for high res videos (>720p in the case of the Galaxy S4 mini) so you have to properly release one instance of MediaCodec before starting the other one, something like this:

  public void releaseResources() {
      // release everything we grabbed
      if (mDecoder != null) {
          try {
              mDecoder.stop();
              mDecoder.release();
              mDecoder = null;
          } catch (Exception e) {
              Timber.d("releaseResources(): message %s cause %s", e.getMessage(),  e.getCause());
              e.printStackTrace();
          }
      }
      if (mExtractor != null) {
          try {
              mExtractor.release();
              mExtractor = null;
          } catch (Exception e) {
              Timber.d("releaseResources(): message %s cause %s", e.getMessage(),  e.getCause());
              e.printStackTrace();
          }
      }
  }
  

• Secondly, why the MoviePlayer fails to configure for 1080p videos. In my case, since I start this activity from a media picker activity with a VideoView for preview which has a MediaPlayer.OnErrorListener() registered to detect faulty video. The problem is the onError() callback has some really weird behavior on the Galaxy S4 mini while it work perfectly fine on my development devices. My wild guess this MediaPlayer.OnErrorListener() leaves the MediaCodec in a bad state that cause a lot of headache later on.

  So the solution is to set a blank OnErrorListener() and properly release the VideoView before doing anything else with MediaCodec. Something like this:

  mVideoView.setOnErrorListener(new MediaPlayer.OnErrorListener() {
          @Override
          public boolean onError(MediaPlayer mp, int what, int extra) {
              return true;
          }
  });
  .....
  mVideoView.stopPlayback();
  doOtherThingWithMediaCodec();
  

This explain a lot of weird thing that happen on different devices. I hope this help someone else debug their app.