C++ OpenCV imdecode slow

I send a byte array of an image from C# to a C++ Library. I decode the image with OpenCV (Version 3.3.1) BMP images are fast in decoding but JPEG images are slow.

How i can speed up the decoding time for JPEG images? (Multithreading, GPU, ...?)

Performance of decode

---------------------------------------------------------
| Resolution | Format | Size  | Duration  |             |
---------------------------------------------------------
| 800x600    | BMP    | 2MB   | 0.7 ms    |             |
---------------------------------------------------------
| 800x600    | JPEG   | 10KB  | 4 ms      | 500% slower |
---------------------------------------------------------

OpenCV C++ Method

VMAPI char* __stdcall SendImage(unsigned char* pArray, int nSize)
{
    cv::Mat buf(1, nSize, CV_8UC1, (void*)pArray);
    auto start = std::chrono::high_resolution_clock::now();
    //cv::Mat input = cv::imdecode(buf, CV_LOAD_IMAGE_COLOR);
    cv::Mat input = cv::imdecode(buf, -1);
    auto finish = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double> elapsed = finish - start;
    std::string result = "Test Version 1.0 - Elapsed time: " + std::to_string(elapsed.count() * 1000) + " s\n";

    return _strdup(result.c_str());
}

C# Request

[DllImport("VideoModule.dll")]
public static extern string SendImage(IntPtr pArray, int nSize);

static void ProcessImage()
{
    var bitmap = new Bitmap(800, 600);
    using (var graphic = Graphics.FromImage(bitmap))
    {
        graphic.Clear(Color.White);
        graphic.DrawRectangle(new Pen(Color.DarkBlue), 20, 20, 60, 60);
        graphic.DrawRectangle(new Pen(Color.DarkGreen), 200, 200, 60, 60);
        graphic.DrawRectangle(new Pen(Color.Red), 500, 400, 60, 60);
    }

    var memoryStream = new MemoryStream();
    //Return an image in JPEG
    bitmap.Save(memoryStream, ImageFormat.Jpeg);
    //Return an image in BMP
    //bitmap.Save(memoryStream, ImageFormat.Bmp);
    var imageData = memoryStream.GetBuffer();

    var size = Marshal.SizeOf(imageData[0]) * imageData.Length;
    IntPtr pnt = Marshal.AllocHGlobal(size);

    try
    {
        // Copy the array to unmanaged memory.
        Marshal.Copy(imageData, 0, pnt, imageData.Length);
    }
    catch (Exception)
    {
    }

    result = SendImage(pnt, imageData.Length);
    Marshal.FreeHGlobal(pnt);
    Console.WriteLine(result);
}

You can use a other codec for example