Windows OCR engine fails to recognize the text in

2019-03-04 17:20发布

问题:

I have a cordova project where I have a "scribble pad" where the user can scribble their notes. This is a simple canvas object, and I'd like to get the OCR Engine to convert it into text. I'm struggling to convert the canvas data into the software bitmap that OCR Engine supports.

All the samples are based either around loading a file from the storage or reading a stream from camera. Do I have to save this canvas into a file on a device and read it back in into a stream?

I'd welcome the guidance in here as images are something I struggle with.

[Update]

So, I've managed to somehow get the stream, but unfortunately, OCR is not recognizing it.

I have the canvas object and after page is loaded, I place the text into it, so any capable OCR should be able to read it. I also have the "img" element, for checking whether the stream is correct and contains the correct bitmap. Here is the code that handles the convas conversion to OCR recognition

     var blob = canvas.msToBlob();

// This is the stream I'll use for OCR detection
                var randomAccessStream = blob.msDetachStream();

// This is the stream I'll use for the image element to make sure the stream above contains what I've placed into the canvas    
                var blob2 = MSApp.createBlobFromRandomAccessStream("image/png", randomAccessStream.cloneStream());
                // Angular JS scope model
                $scope.imageUrl = URL.createObjectURL(blob2);


// This works, but returns ""
                var scope = this;
                if (!this.ocrEngine)
                    return;

                var bitmapDecoder = Windows.Graphics.Imaging.BitmapDecoder;
                bitmapDecoder.createAsync(randomAccessStream).then(function (decoder) {
                    return decoder.getSoftwareBitmapAsync();
                }).then(function (bitmap) {
                    return scope.ocrEngine.recognizeAsync(bitmap);
                }).then(function (result) {
                    console.log(result.text);
                }); 

After this all runs, the image is given the src and is loaded and contains exactly whatever is in the canvas so the stream is correct.

The ocrEngine is setup the following way:

var Globalization = Windows.Globalization;
            var OCR = Windows.Media.Ocr;
            this.ocrEngine = OCR.OcrEngine.tryCreateFromUserProfileLanguages();
            if (!this.ocrEngine) {
                // Try to create OcrEngine for specified language.
                // If language is not supported on device, method returns null.
                this.ocrEngine = OCR.OcrEngine.tryCreateFromLanguage(new Globalization.Language("en-us"));
            }
            if (!this.ocrEngine) {
                console.error("Selected language is not available.");
            }

Why is OCR not recognizing simple 'Hello World' ?

回答1:

well, that was rather embarrassing to realize that the reason why the OCR failed to read anything, even a system written text was that the resulting, generated image had a transparent background. Once I've included a rectangle with the white fill it all started to work correctly.

Unfortunately, the OCR is struggling to recognize anything I scribble on the canvas, so e.g. handwritten numbers or multiline text in canvas are not being recognized, see below

Recognized:

Not recognized

Not recognized:

Not recognized:

Then I've found Windows.UI.Input.Inking namespace and I reckon that's the only way to go.