Replace the camera stream in a Wikitude ArchitectV

问题:

I'm working on an Android mobile app oriented to the real time augmented visualization of a drone's camera view (specifically I'm working on a DJI Phantom 3 Professional with relative SDK). I'm using Wikitude framework for the AR part. Every Wikitude sample works by augmenting the smartphone's camera view, so I need to redirect the input video stream. At this stage, by using the DJI SDK features, I have a TextureView object containing the correctly decoded stream coming from the drone. The following are the relative instructions on the top of the current view's .xml file:

<TextureView
    android:id="@+id/video_previewer_surface"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_gravity="center"
    android:layout_centerHorizontal="true"
    android:layout_above="@+id/linearLayout" />

Actually, Wikitude Architect View is composed by the rendering of the camera stream and the rendering of the augmentation, as depicted in the following figure: Architect View Composition

sample_cam.xml file:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" 
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="horizontal">

   <com.wikitude.architect.ArchitectView
      android:id="@+id/architectView"
      android:layout_width="fill_parent"
      android:layout_height="fill_parent"/>
</LinearLayout>

Is there a way to replace the current camera stream of the Wikitude Architect View with the content of my TextureView?

Alternatively, is there a way to make the camera stream of the ArchitectView as a transparent background, in order to overlap this layer to the TextureView one?

The following could be a portion of the code of the .xml final file:

<TextureView
    android:id="@+id/video_previewer_surface"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_gravity="center"
    android:layout_centerHorizontal="true"
    android:layout_above="@+id/linearLayout" />
<com.wikitude.architect.ArchitectView  // with transparent background for camera
   android:id="@+id/architectView"
   android:layout_width="fill_parent"
   android:layout_height="fill_parent"/>

回答1:

You could implement your own Wikitude Input Plugin in combination with dji's Video Stream Decoding. So you get the raw video data from the dji sdk and pass it to the wikitude sdk. A sample code for the Input Plugin can be found in the wikitude sample app with the sample name "Custom Camera".

回答2:

Alex answer is correct and it works. Here are some more details: The Wikitude Input Plugin can be used to supply your own camera input and your can also apply your own augmentations and your own rendering. See The Wikitude Input Plugin documentation.

According to the Wikitude samples, the Custom Camera Plugin can be used as a starting point. You develop your own Camera Extension and within the constructor, you do something like this:

    arVideoView = new SurfaceViewRenderer(activity);
    arVideoView.setId(R.id.local_surface_rendeer);
    LinearLayout.LayoutParams param1 = new LinearLayout.LayoutParams(
            FrameLayout.LayoutParams.MATCH_PARENT,
            FrameLayout.LayoutParams.MATCH_PARENT,
            1.0f
    );
    arVideoView.setLayoutParams(param1);
    arVideoView.setKeepScreenOn(true);
    architectView.addView(arVideoView);

In this case I added a WebRTC camera view to the Architect View.

Then I implemented a callback for the incoming frames.

// Implementation detail: bridge the VideoRenderer.Callbacks interface to the
// VideoStreamsView implementation.
public class VideoCallbacks implements VideoRenderer.Callbacks {
    @Override
    public void renderFrame(final VideoRenderer.I420Frame i420Frame) {
        Log.v(TAG, "renderFrame ..."+i420Frame);

.. convert the frames to nv21 Log.v(TAG, "make byte array"); byte[] b = getNV21(0,0, bmpRotate.getWidth(), bmpRotate.getHeight(), bmpRotate);

// and pass the frame to the plugin for further processing notifyNewCameraFrameN21(b);

                gotFrame = false;
            }
        });

    }
}

Hope this helps. Happy coding! Phil