TangoService_connectOnFrameAvailable() gets stuck

Question

There seems to be an issue concering the reception of color frames in the Leibniz Release 1.10 as well: when having registered a callback using TangoService_connectOnFrameAvailable(TANGO_CAMERA_COLOR,NULL,onFrameAvailable) the callback onFrameAvailable() will either never ever be called or TangoService_connectOnFrameAvailable() crashes with the following error:

04-20 13:29:44.384: E/tango_client_api(4712): TangoErrorType TangoService_connectOnFrameAvailable(TangoCameraId, void*, void ()(void, TangoCameraId, const TangoImageBuffer*)): Internal Error: connectSurface(), cam id 0, failed internally.

The release notes say

[...] config_enable_color_camera has been added to the configuration flags. We recommend you always explicitly set this flag to true if accessing the color camera. You must set the flag true for TangoService_connectOnFrameAvailable() or TangoService_connectTextureId() to succeed after TangoService_connect() is called. [...]

Thus, if I set that flag to true between the calls of TangoService_connect() and TangoService_connectOnFrameAvailable(), the callback onFrameAvailable() will never ever be called, if I set that flag to true before TangoService_connect() TangoService_connectOnFrameAvailable() will always crash.

Thus, what am I doing wrong? Is there a code snippet available or something? That would be really helpful... Unfortunately, none of the examples use color frames...

Man, after having had similar problems with the Kalman Release 1.9, I begin to wonder if the SDKs are thoroughly tested before being released in the first place...

Answer 1

Here is my code for converting an NV21 to an RGB frame. Maybe it is of any use...

static void
  cb_onFrameAvailable
  (
    void*                     contextA,
    TangoCameraId             idA,
    const TangoImageBuffer*  imageBufferA
  )
{
  // --- local constants ------------------------------

  // image width and height
  const int W               = imageBufferA->width;
  const int H               = imageBufferA->height;

  // sizes of Y, U, and V pixel arrays.
  const int sizeOfYDataL   = W * H;

  // indices, marking the begin of the y, u, and v data in the pixel buffer.
  const int beginOfYDataL  = 0;
  const int beginOfUVDataL  = sizeOfYDataL;

  // YUV, Y, and UV pixel sub arrays.
  const byte*  yuvArrL     = imageBufferA->data;
  const byte*  yArrL       = &yuvArrL[ beginOfYDataL  ];
  const byte*  uvArrL      = &yuvArrL[ beginOfUVDataL ];

  // --- local variables ------------------------------

  // image pixel coordinates.
  int xL,yL;

  // halved image pixel coordinates.
  int hxL,hyL;

  // ARGB value.
  int argbL;

  // --------------------------------------------------

  // translate YUV NV21 -> ARGB, using
  //
  //      / R \   / 1.000   0.000   1.596 \   /   Y   \
  //      | G | = | 1.000  -0.391  -0.813 | * | U-128 |
  //      \ B /   \ 1.000   2.018   0.000 /   \ V-128 /
  //

  // Note: start value yL=1 as the first scan line of the color image is ..
  //       .. reserved for metadata instead of image pixels.

  for( yL=1,hyL=0; yL<H; yL++,hyL=yL>>1 )
  {
    for( xL=0,hxL=0; xL<W; xL++,hxL=xL>>1 )
    {
      const int y = static_cast<int>( yArrL [  yL*W +    xL   ] )      ;
      const int v = static_cast<int>( uvArrL[ hyL*W + 2*hxL   ] ) - 128;
      const int u = static_cast<int>( uvArrL[ hyL*W + 2*hxL+1 ] ) - 128;

      int R = static_cast<int>( y               + ( 1.596f*v) );
      int G = static_cast<int>( y + (-0.391f*u) + (-0.813f*v) );
      int B = static_cast<int>( y + ( 2.018f*u)               );

      // clip RGB values to [0..255].
      R = R < 0 ? 0 : (R > 255 ? 255 : R);
      G = G < 0 ? 0 : (G > 255 ? 255 : G);
      B = B < 0 ? 0 : (B > 255 ? 255 : B);

      // combine to ARGB value.
      argbL = 0xff000000 | (R << 16) | (G << 8) | B;
    } // for
  } // for
} // function

Answer 2

Alright assuming that the problem is not what I mentioned in the comments section. Here is code snippet testing the onFrameAvailable Callback.

Note: I have modified the HelloTangoJni Example from the Tango-examples-c repository for this.

In TangoHandler.h add

 TangoErrorType ConnectYUVFrameCallback();

Modify TangoHandler.cc

TangoErrorType TangoHandler::SetupConfig() {
  // TANGO_CONFIG_DEFAULT is enabling Motion Tracking and disabling Depth
  // Perception.
  tango_config_ = TangoService_getConfig(TANGO_CONFIG_DEFAULT);
  if (tango_config_ == nullptr) {
  return TANGO_ERROR;
  }
  TangoConfig_setBool(tango_config_,"config_enable_color_camera",true);
  return TANGO_SUCCESS;
}


TangoErrorType TangoHandler::ConnectYUVFrameCallback() {
    TangoErrorType onFrameErrorType=TangoService_connectOnFrameAvailable( TANGO_CAMERA_COLOR, NULL, onFrameAvailable);
    if( onFrameErrorType!= TANGO_SUCCESS)
    {
         LOGI("GOOGLE TANGO ONFRAMEAVAILABLE FAILED!");
    }
    LOGI("GOOGLE TANGO ONFRAMEAVAILABLE SUCCESS!");
    return onFrameErrorType;
}

static void onFrameAvailable( void* context, const TangoCameraId id, const TangoImageBuffer* buffer )
{
  int width = buffer->width;
  int height = buffer->height;
  LOGI("width and height is: %d,%d",width,height);
}

In TangoNative.cc add

JNIEXPORT jint JNICALLJava_com_projecttango_experiments_nativehellotango_TangoJNINative_connectOnFrameAvailableCallback(
JNIEnv*, jobject) 
{
    return static_cast<int>(tango_handler.ConnectYUVFrameCallback());
}

In TangoJNINative.java add

// Connect the onFrameAvailable callback.
public static native int connectOnFrameAvailableCallback();

In HelloTangoActivity.java modify onResume()

protected void onResume() {
   super.onResume();
   // Setup Tango configuraturation.
   TangoJNINative.setupConfig();
   int status = 0;
   TangoJNINative.connect();
   status = TangoJNINative.connectOnFrameAvailableCallback();
   mIsTangoServiceConnected = true;
}