convert QVideoFrame to QImage

Question

I want to get every frames from a QMediaPlayer and convert it to QImage (or cv::Mat)

so I used videoFrameProbed signal from QVideoProbe:

connect(&video_probe_, &QVideoProbe::videoFrameProbed, 
         [this](const QVideoFrame& currentFrame){
   //QImage img = ??
}

But I didn't find any way for getting QImage from QVideoFrame!

How can I convert QVideoFrame to QImage ?!

Answer 1

You can use QImage's constructor:

 QImage img( currentFrame.bits(),
             currentFrame.width(),
             currentFrame.height(),
             currentFrame.bytesPerLine(),
             imageFormat);

Where you can get imageFormat from pixelFormat of the QVideoFrame:

 QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(currentFrame.pixelFormat());

Answer 2

Qt has a private function for converting QVideoFrame to QImage. If you want to use it, you'll need to header like a forward declaration:

QImage qt_imageFromVideoFrame( const QVideoFrame& f );

The implementation uses QVideoFrame::map() and does low-level manipulation with the bits. The converters handle a variety of conditions including YUV.

The qt_imageFromVideoFrame works on almost all platforms. i.e. Windows, macOS, linux and iOS. The only platform it doesn't work reliably well is Android. There, you'll need to use OpenGL calls to extract the VideoFrame.

    if ( videoFrame.handleType() == QAbstractVideoBuffer::GLTextureHandle )
    {
        QImage image( videoFrame.width(),  videoFrame.height(), QImage::Format_ARGB32 );
        GLuint textureId = static_cast<GLuint>( videoFrame.handle().toInt() );
        QOpenGLContext* ctx = QOpenGLContext::currentContext();
        QOpenGLFunctions* f = ctx->functions();
        GLuint fbo;
        f->glGenFramebuffers( 1, &fbo );
        GLint prevFbo;
        f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo );
        f->glBindFramebuffer( GL_FRAMEBUFFER, fbo );
        f->glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureId, 0 );
        f->glReadPixels( 0, 0,  videoFrame.width(),  videoFrame.height(), GL_RGBA, GL_UNSIGNED_BYTE, image.bits() );
        f->glBindFramebuffer( GL_FRAMEBUFFER, static_cast<GLuint>( prevFbo ) );
        return image;
    }

I have an implementation of QImage QVideoFrameToQImage( const QVideoFrame& videoFrame ); on GitHub:

https://github.com/stephenquan/MyVideoFilterApp/blob/master/QVideoFrameToQImage.cpp

Answer 3

For QCamera output, that method doesn't always work. In particular, QVideoFrame::imageFormatFromPixelFormat() returns QImage::Format_Invalid when given QVideoFrame::Format_Jpeg, which is what's coming out of my QCamera. But this works:

QImage Camera::imageFromVideoFrame(const QVideoFrame& buffer) const
{
    QImage img;
    QVideoFrame frame(buffer);  // make a copy we can call map (non-const) on
    frame.map(QAbstractVideoBuffer::ReadOnly);
    QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(
                frame.pixelFormat());
    // BUT the frame.pixelFormat() is QVideoFrame::Format_Jpeg, and this is
    // mapped to QImage::Format_Invalid by
    // QVideoFrame::imageFormatFromPixelFormat
    if (imageFormat != QImage::Format_Invalid) {
        img = QImage(frame.bits(),
                     frame.width(),
                     frame.height(),
                     // frame.bytesPerLine(),
                     imageFormat);
    } else {
        // e.g. JPEG
        int nbytes = frame.mappedBytes();
        img = QImage::fromData(frame.bits(), nbytes);
    }
    frame.unmap();
    return img;
}