Trying to render javafx WebView to offscreen buffe

Question

The ultimate goal is to be able to record the output of a WebView at 30fps or better, perhaps by setting up an FBO for javafx? I could then pull out frames at whatever framerate I wanted.

I've poked around some and I came across UploadingPainter in ViewScene, which makes me think that this is possible. The struggle is that this is seemingly under the hood and somewhat new to me.

Anyone know of a way to make something like this work?

This is the code that I came across during debugging:

@Override
public void setStage(GlassStage stage) {
    super.setStage(stage);
    if (stage != null) {
        WindowStage wstage  = (WindowStage)stage;
        if (wstage.needsUpdateWindow() || GraphicsPipeline.getPipeline().isUploading()) {
            if (Pixels.getNativeFormat() != Pixels.Format.BYTE_BGRA_PRE ||
                ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
                throw new UnsupportedOperationException(UNSUPPORTED_FORMAT);
            }
            painter = new UploadingPainter(this);
        } else {
            painter = new PresentingPainter(this);
        }
        painter.setRoot(getRoot());
        paintRenderJob = new PaintRenderJob(this, PaintCollector.getInstance().getRendered(), painter);
    }
}

Answer 1

Here is an example of capturing an animation in a WebView.

The images captured from the web view are placed in a Paginator for viewing purposes just so that it is easy to review them. You could use SwingFXUtils and ImageIO to write them out to files instead if you wish. If you want to get the resultant images into a buffer, you could use their PixelReader.

It doesn't quite work the way I wanted it to. I wanted to snapshot the WebView without placing it in a visible stage. Taking snapshots of nodes that are not in a Stage works fine for every other node type in JavaFX (as far as I know), however, for some weird reason, it does not work for WebView. So the sample actually creates a new Stage behind the display window that displays the image sequence for the animation capture result. I'm aware that not exactly what you want, but it is what it is...

import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.beans.property.*;
import javafx.collections.*;
import javafx.concurrent.Worker;
import javafx.geometry.Insets;
import javafx.scene.Scene;
import javafx.scene.SnapshotParameters;
import javafx.scene.control.*;
import javafx.scene.image.*;
import javafx.scene.layout.*;
import javafx.scene.web.WebView;
import javafx.stage.Stage;

public class WebViewAnimationCaptor extends Application {

    private static final String CAPTURE_URL =
            "https://upload.wikimedia.org/wikipedia/commons/d/dd/Muybridge_race_horse_animated.gif";

    private static final int N_CAPS_PER_SECOND = 10;
    private static final int MAX_CAPTURES = N_CAPS_PER_SECOND * 5;
    private static final int W = 186, H = 124;

    class CaptureResult {
        ObservableList<Image> images = FXCollections.observableArrayList();
        DoubleProperty progress = new SimpleDoubleProperty();
    }

    @Override public void start(Stage stage) {
        CaptureResult captures = captureAnimation(CAPTURE_URL);
        Pane captureViewer = createCaptureViewer(captures);

        stage.setScene(new Scene(captureViewer, W + 40, H + 80));
        stage.show();
    }

    private StackPane createCaptureViewer(CaptureResult captures) {
        ProgressIndicator progressIndicator = new ProgressIndicator();
        progressIndicator.progressProperty().bind(captures.progress);
        progressIndicator.setPrefSize(W, H);

        StackPane stackPane = new StackPane(progressIndicator);
        stackPane.setPadding(new Insets(10));
        if (captures.progress.get() >= 1.0) {
            stackPane.getChildren().setAll(
                createImagePages(captures.images)
            );
        } else {
            captures.progress.addListener((observable, oldValue, newValue) -> {
                if (newValue.doubleValue() >= 1.0) {
                    stackPane.getChildren().setAll(
                            createImagePages(captures.images)
                    );
                }
            });
        }

        return stackPane;
    }

    private Pagination createImagePages(ObservableList<Image> captures) {
        Pagination pagination = new Pagination();
        pagination.setPageFactory(param -> {
            ImageView currentImage = new ImageView();
            currentImage.setImage(
                    param < captures.size()
                            ? captures.get(param)
                            : null
            );

            StackPane pageContent = new StackPane(currentImage);
            pageContent.setPrefSize(W, H);

            return pageContent;
        });

        pagination.setCurrentPageIndex(0);
        pagination.setPageCount(captures.size());
        pagination.setMaxPageIndicatorCount(captures.size());

        return pagination;
    }

    private CaptureResult captureAnimation(final String url) {
        CaptureResult captureResult = new CaptureResult();

        WebView webView = new WebView();
        webView.getEngine().load(url);
        webView.setPrefSize(W, H);

        Stage captureStage = new Stage();
        captureStage.setScene(new Scene(webView, W, H));
        captureStage.show();

        SnapshotParameters snapshotParameters = new SnapshotParameters();
        captureResult.progress.set(0);

        AnimationTimer timer = new AnimationTimer() {
            long last = 0;

            @Override
            public void handle(long now) {
                if (now > last + 1_000_000_000.0 / N_CAPS_PER_SECOND) {
                    last = now;
                    captureResult.images.add(webView.snapshot(snapshotParameters, null));
                    captureResult.progress.setValue(
                            captureResult.images.size() * 1.0 / MAX_CAPTURES
                    );
                }

                if (captureResult.images.size() > MAX_CAPTURES) {
                    captureStage.hide();
                    this.stop();
                }
            }
        };

        webView.getEngine().getLoadWorker().stateProperty().addListener((observable, oldValue, newValue) -> {
            if (Worker.State.SUCCEEDED.equals(newValue)) {
                timer.start();
            }
        });

        return captureResult;
    }

    public static void main(String[] args) { launch(args); }
}

To fine-tune the animation sequence capture, you could review this info on AnimationTimers in JavaFX.

If you need to make the thing "headless", so that a visible stage is not required, you could try this gist by danialfarid which performs "Java Image Capture, HTML Snapshot, HTML to image" (though I did not write the linked gist and have not tried it).

---

Headless is key, in my case. The (linux) machines in question run in a server farm totally headless. As for the gist, I see a show() in there but I'll take a closer look to make sure that I didn't overlook something.

The gist is based upon the Monocle glass rendering toolkit for JavaFX systems. This toolkit supports software based headless rendering on any system.

From the Monocle Documentation:

The headless port does nothing. It is for when you want to run JavaFX with no graphics, input or platform dependencies. Rendering still happens, it just doesn’t show up on the screen.

The headless port uses the LinuxInputDeviceRegistry implementation of InputDeviceRegistry. However the headless port does not access any actual Linux devices or any native APIs at all; it uses the Linux input registry in device simulation mode. This allows Linux device input to be simulated even on non-Linux platforms. The tests in tests/system/src/test/java/com/sun/glass/ui/monocle/input make extensive use of this feature.

If the JavaFX Monocle based approach ends up not working out for you, you could consider another (not JavaFX related) headless HTML rendering kit, such as PhantomJS.