How do I obtain pixel data from a canvas I don'

Question

I am trying to get the pixel RGBA data from a canvas for further processing. I think the canvas is actually a Unity game if that makes a difference.

I am trying to do this with the canvas of the game Shakes and Fidget. I use the readPixels method from the context.

This is what I tried:

var example = document.getElementById('#canvas');
var context = example.getContext('webgl2');      // Also doesn't work with: ', {preserveDrawingBuffer: true}'
var pixels = new Uint8Array(context.drawingBufferWidth * context.drawingBufferHeight * 4); 
context.readPixels(0, 0, context.drawingBufferWidth, context.drawingBufferHeight, context.RGBA, context.UNSIGNED_BYTE, pixels);

But all pixels are black apparently (which is not true obviously).

Edit: Also, I want to read the pixels multiple times. Thanks everyone for your answers. The answer provided by @Kaiido worked perfectly for me :)

Answer 1

You can require a Canvas context only once. All the following requests will either return null, or the same context that has been created before if you passed the same options to getContext().

Now, the one page you linked to didn't pass the preserveDrawingBuffer option when creating their context, which means that to be able to grab the pixels info from there, you will have to hook up in the same event loop as the one the game loop occur.
Luckily, this exact game does use a simple requestAnimationFrame loop, so to hook up to the same event loop, all we need to do is to also wrap our code in a requestAnimationFrame call.

Since callbacks are stacked, and that they do require the next frame from one such callback to create a loop, we can be sure our call will get stacked after their.

I now realize it might not be obvious, so I'll try to explain further what requestAnimationFrame does, and how we can be sure our callback will get called after Unity's one.

requestAnimationFrame(fn) pushes fn callback into a stack of callbacks that will all get called at the same time in First-In-First-Out order, just before the browser will perform its paint to screen operations. This happens once in a while (generally 60 times per second), at the end of the closest event loop.
It can be understood as a kind of setTimeout(fn , time_remaining_until_next_paint), with the main difference that it is guaranteed that requestAnimationFrame callback executor will get called at the end of the event loop, and thus after other js execution of this event loop.
So if we were to call requestAnimationFrame(fn) in the same event loop that the one where the callbacks will get called, our fake time_remaining_until_next_paint would be 0, and fn will get pushed at the bottom of our stack (last in, last out).
And when calling requestAnimationFrame(fn) from inside the callbacks executor itself, time_remaining_until_next_paint would be something around 16, and fn will get called among the first ones at the next frame.

So any calls to requestAnimationFrame(fn) made from outside of the requestAnimationFrame's callbacks executor is guaranteed to be called in the same event loop than a requestAnimationFrame powered loop, and to be called after.

So all we need to grab these pixels, is to wrap the call to readPixels in a requestAnimationFrame call, and to call it after Unity's loop started.

var example = document.getElementById('#canvas');
var context = example.getContext('webgl2') || example.getContext('webgl');
var pixels = new Uint8Array(context.drawingBufferWidth * context.drawingBufferHeight * 4);
requestAnimationFrame(() => {
  context.readPixels(0, 0, context.drawingBufferWidth, context.drawingBufferHeight, context.RGBA, context.UNSIGNED_BYTE, pixels);
  // here `pixels` has the correct data
});

Answer 2

Likely you either need to read the pixels in the same event as they are rendered, or you need to force the canvas to use preserveDrawingBuffer: true so you can read the canvas at any time.

To do the second override getContext

HTMLCanvasElement.prototype.getContext = function(origFn) {
  const typesWeCareAbout = {
    "webgl": true,
    "webgl2": true,
    "experimental-webgl": true,
  };
  return function(type, attributes = {}) {
    if (typesWeCareAbout[type]) {
      attributes.preserveDrawingBuffer = true;
    }
    return origFn.call(this, type, attributes);
  };
}(HTMLCanvasElement.prototype.getContext);

Put that at the top of the file before the Unity game OR put it in a separate script file and include it before the Unity game.

You should now be able to get a context on whatever canvas Unity made and call gl.readPixels anytime you want.

For the other method, getting pixels in the same event, you would instead wrap requestAnimationFrame so that you can insert your gl.readPixels after Unity's use of requestAnimationFrame

window.requestAnimationFrame = function(origFn) {
  return function(callback) {
    return origFn(this, function(time) {
      callback(time);
      gl.readPixels(...);
    };
  };
}(window.requestAnimationFrame);

Another solution would be to use a virtual webgl context. This library shows an example of implementing a virtual webgl context and shows an example of post processing the unity output

note that at some point Unity will likely switch to using an OffscreenCanvas. At that point it will likely require other solutions than those above.