How do I grab frames from a video stream on Window

Question

I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.

There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.

Here's a short piece of code from where I try to do that (project itself attached at the bottom).

    ComPtr<ID3D11Texture2D> spTextureDst;
    MEDIA::ThrowIfFailed(
        m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
        );

    auto rcNormalized = MFVideoNormalizedRect();
    rcNormalized.left = 0;
    rcNormalized.right = 1;
    rcNormalized.top = 0;
    rcNormalized.bottom = 1;
    MEDIA::ThrowIfFailed(
        m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
        );

    //copy the render target texture to the readable texture.
    m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
    m_spDX11DeviceContext->Flush();

    //Map the readable texture;                 
    D3D11_MAPPED_SUBRESOURCE mapped = {0};
    m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
    void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
    memcpy(buffer, mapped.pData,600 * 400 * 3);
    //unmap so we can copy during next update.
    m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);


    // and the present it to the screen
    MEDIA::ThrowIfFailed(
        m_spDX11SwapChain->Present(1, 0)
        );            
}

The error I get is:

First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057

I am not really sure how to pursue it further it since, like I said, there's very little docs about it.

Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).

Answer 1

UPDATE success!! see edit at bottom

---

Some partial success, but maybe enough to answer your question. Please read on.

On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.

So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.

So my first change was in function CreateDX11Device() as follows:

static const D3D_FEATURE_LEVEL levels[] = {
/*
        D3D_FEATURE_LEVEL_11_1,
        D3D_FEATURE_LEVEL_11_0,  
        D3D_FEATURE_LEVEL_10_1,
        D3D_FEATURE_LEVEL_10_0,
*/
        D3D_FEATURE_LEVEL_9_3,
        D3D_FEATURE_LEVEL_9_2,
        D3D_FEATURE_LEVEL_9_1
    };

Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.

More Googling led to my second discovery -

Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.

Anyway, I now used this code to get the video frame:

ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);

After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.

So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.

        //copy the render target texture to the readable texture.
        m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
        m_spDX11DeviceContext->Flush();

        //Map the readable texture;                 
        D3D11_MAPPED_SUBRESOURCE mapped = {0};
        HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
        void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
        memcpy(buffer, mapped.pData,176 * 144 * 3);
        //unmap so we can copy during next update.
        m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);

Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.

Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.

EDIT

Changes made to texture description in FramePlayer::CreateBackBuffers()

        //make first texture cpu readable
        D3D11_TEXTURE2D_DESC texDesc = {0};
        texDesc.Width = 176;
        texDesc.Height = 144;
        texDesc.MipLevels = 1;
        texDesc.ArraySize = 1;
        texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
        texDesc.SampleDesc.Count = 1;
        texDesc.SampleDesc.Quality =  0;
        texDesc.Usage = D3D11_USAGE_STAGING;
        texDesc.BindFlags = 0;
        texDesc.CPUAccessFlags =  D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
        texDesc.MiscFlags = 0;

        MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));

Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:

void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test

SUCCESS

I have now succeeded in saving an individual frame, but now without the use of the copy texture.

First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.

Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp

#include "..\DirectXTex\DirectXTex.h"  // nb - use the relative path you copied to
#include <wincodec.h>

Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name

// new frame available at the media engine so get it 
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));

auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));

// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
    // get the image object from the wrapper
    const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);

    // set some place to save the image frame
    StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
    Platform::String ^szPath = dataFolder->Path + "\\frame.png";

    // save the image to file
    hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}

// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));            

I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))

Can you take a fresh look and update your results in comments?

Answer 2

I think OpenCV may help you. OpenCV offers api to capture frames from camera or video files. You can download it here http://opencv.org/downloads.html. The following is a demo I writed with "OpenCV 2.3.1".

#include "opencv.hpp"

using namespace cv;
int main()
{
    VideoCapture cap("demo.avi");   // open a video to capture
    if (!cap.isOpened())        // check if succeeded
        return -1;

    Mat frame;
    namedWindow("Demo", CV_WINDOW_NORMAL);
    // Loop to capture frame and show on the window
    while (1) {
        cap >> frame;
        if (frame.empty())
            break;

        imshow("Demo", frame);
        if (waitKey(33) >= 0)  // pause 33ms every frame
            break;
    }

    return 0;
}

Answer 3

Look at the Video Thumbnail Sample and the Source Reader documentation.

You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail