Why does AVSampleBufferDisplayLayer fail with Oper

I'm using an AVSampleBufferDisplayLayer to decode and display H.264 video streamed from a server. When my app goes into the background and then returns to the foreground, the decoding process gets screwed up and the AVSampleBufferDisplayLayer fails. The error I'm seeing is:

H.264 decoding layer has failed: Error Domain=AVFoundationErrorDomain
  Code=-11847 "Operation Interrupted" UserInfo=0x17426c500
  {NSUnderlyingError=0x17805fe90 "The operation couldn’t be completed.
    (OSStatus error -12084.)",
   NSLocalizedRecoverySuggestion=Stop other operations and try again.,
   NSLocalizedDescription=Operation Interrupted}

Has anybody else run into issues like this with AVSampleBufferDisplayLayer? What does this mean?

I have tried destroying the AVSampleBufferDisplayLayer and creating a new one when I get the error, but then I start receiving other errors from the H.264 decoder:

Error Domain=AVFoundationErrorDomain Code=-11821 "Cannot Decode"
UserInfo=0x1740e9700 {AVErrorMediaSubTypeKey=(1635148593),
  NSLocalizedFailureReason=The media data could not be decoded. It may be damaged.,
  NSUnderlyingError=0x174247680 "The operation couldn’t be completed. (OSStatus error -12909.)",
  AVErrorMediaTypeKey=vide,
  AVErrorPresentationTimeStampKey=CMTime: {7/30 = 0.233},
  NSLocalizedDescription=Cannot Decode}

I was not receiving any of those errors before the AVSampleBufferDisplayLayer failed.

After you rebuild a new AVSampleBufferDisplayLayer, you should enqueue it with the last nearest IDR frame except current frame is the IDR, which means, you should save nalus in one GOP when decoding and delete them when next IDR is coming.