Testing framework#

The webgpu.testing module provides reusable pytest infrastructure for visual regression tests that run against a real WebGPU device inside headless Chrome. It is designed so that downstream packages (such as ngsolve_webgpu) can set up their own test suites with minimal boilerplate.

Architecture#

Tests run inside a Docker container that provides:

  • Headless Chrome with WebGPU enabled

  • Lavapipe (Mesa software Vulkan) so no physical GPU is required

  • Playwright to drive the browser from Python

The webgpu.testing module then:

  1. Launches a websocket server (webgpu.platform.init)

  2. Serves a minimal HTML page that connects back to Python

  3. Provides pytest fixtures that give tests access to the live WebGPU device, a browser page and helper methods for screenshots and baseline comparison

Docker images#

A three-layer Docker image scheme keeps things DRY:

Dockerfile.base (provided by webgpu)

Playwright + Chrome + lavapipe + Python test dependencies. Clean /app work directory, no source code or tests. Downstream packages (including webgpu’s own test image) derive from this image.

Dockerfile (webgpu tests)

Extends the base image, installs the webgpu package from source and copies the test suite into /app/tests.

Downstream Dockerfile (e.g. ngsolve_webgpu)

Extends the base image, installs webgpu (from PyPI or source), additional dependencies and the downstream package, copies its own tests.

┌──────────────────────────────┐
│  Dockerfile.base             │
│  (playwright, chrome,        │
│   lavapipe)                  │
├──────────────┬───────────────┤
│              │               │
│  Dockerfile  │  downstream   │
│  (+ webgpu,  │  Dockerfile   │
│   tests)     │  (+ webgpu,   │
│              │   ngsolve,    │
│              │   own tests)  │
└──────────────┴───────────────┘

Container registry#

The webgpu CI publishes the base image to the GitHub Container Registry when its contents change (only on pushes to main):

ghcr.io/cerbsim/webgpu-base:latest

Because the base image contains only stable system-level dependencies, its layers rarely change and CI builds get fast cache hits. Downstream packages can pull this pre-built image instead of rebuilding from source.

Building locally (from a webgpu checkout):

docker build -f tests/Dockerfile.base -t webgpu-base .

Pulling from the registry:

docker pull ghcr.io/cerbsim/webgpu-base:latest

Using in a downstream Dockerfile:

ARG BASE_IMAGE=ghcr.io/cerbsim/webgpu-base:latest
FROM ${BASE_IMAGE}

# install your package ...

Provided fixtures#

Register the fixtures by adding a single line to your conftest.py:

pytest_plugins = ["webgpu.testing"]

The following fixtures then become available:

browser (session-scoped)

A Playwright Browser instance (headless Chrome with WebGPU flags).

page (function-scoped)

A fresh browser page with no webgpu connection. Useful for pure-JS smoke tests (e.g. checking that navigator.gpu exists).

webgpu_env (session-scoped)

A fully initialised WebGPUTestEnv with a live websocket bridge between Python and the browser. This is the main fixture for rendering tests.

WebGPUTestEnv#

The webgpu_env fixture yields a WebGPUTestEnv instance with the following attributes and methods:

page#

The Playwright Page connected to the websocket bridge.

platform#

The webgpu.platform module (gives access to platform.js).

wj#

The webgpu.jupyter module (patched for headless use).

output_dir#

Path where test output images are written. Must be set by the downstream conftest.py (see below).

baseline_dir#

Path where reference images are stored. Must be set by the downstream conftest.py.

ensure_canvas(width=600, height=600)#

Inject a <canvas> element into the browser page that matches the next wj.Draw() call. Returns the canvas element’s id.

screenshot(name, canvas_id=None)#

Take a Playwright screenshot of a canvas element and save it to output_dir / "{name}.png". Returns the output Path.

readback_texture(scene, path)#

Read back the rendered texture from the GPU via a JS-side buffer readback and save it as a PNG. Returns path.

assert_matches_baseline(scene, filename, *, threshold=0.01)#

Perform a full visual regression check on a rendered scene:

  1. Assert the scene is valid and has render objects

  2. Wait 500 ms for rendering to settle

  3. Read back the GPU texture to output_dir / filename

  4. Compare the output against baseline_dir / filename

Fails the test if more than threshold (fraction) of pixels differ.

When the environment variable UPDATE_BASELINES=1 is set, the output is copied to the baseline instead of compared, making it easy to regenerate references.

Quick start for downstream packages#

This section walks through adding a visual test suite to a package that builds on webgpu. The ngsolve_webgpu test suite is a complete working example of this pattern.

1. Create tests/conftest.py#

"""conftest.py — register webgpu.testing fixtures and set directories."""

from pathlib import Path
import pytest

pytest_plugins = ["webgpu.testing"]

TESTS_DIR = Path(__file__).parent

@pytest.fixture(scope="session", autouse=True)
def _configure_dirs(webgpu_env):
    webgpu_env.output_dir = TESTS_DIR / "output"
    webgpu_env.baseline_dir = TESTS_DIR / "baselines"

This is all the setup needed. Every fixture from webgpu.testing is now available in your tests.

2. Write tests#

"""test_rendering.py"""

class TestMyRendering:
    def test_draw_something(self, webgpu_env):
        # Import your package lazily — see note below.
        from my_package.jupyter import Draw

        webgpu_env.ensure_canvas(600, 600)
        scene = Draw(my_data, width=600, height=600)

        # Readback, validate, and compare against baseline — all in one call
        webgpu_env.assert_matches_baseline(scene, "my_test.png")

Important

Packages that trigger webgpu.jupyter at import time (which calls platform.init() and blocks on a websocket connection) must be imported inside the test function, not at module level. Otherwise pytest will hang during test collection.

3. Create a Dockerfile#

Derive from the webgpu-base image so you get Chrome, lavapipe and Python test dependencies for free. Install webgpu (which includes webgpu.testing) from PyPI:

ARG BASE_IMAGE=ghcr.io/cerbsim/webgpu-base:latest
FROM ${BASE_IMAGE}

RUN pip install --no-cache-dir --break-system-packages webgpu my-dependency

WORKDIR /app
COPY pyproject.toml .
COPY my_package/ my_package/
ENV SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0
RUN pip install --no-cache-dir --break-system-packages .

COPY tests/ tests/

CMD ["pytest", "tests/", "-v", "--tb=short"]

4. Create tests/run_tests.sh#

#!/usr/bin/env bash
set -e
cd "$(dirname "$0")/.."

BASE_IMAGE=webgpu-base
IMAGE=my-package-tests

# Build the webgpu base image from the sibling checkout
WEBGPU_DIR="$(cd ../webgpu && pwd)"
echo "==> Building base image..."
docker build -f "$WEBGPU_DIR/tests/Dockerfile.base" \
    -t "$BASE_IMAGE" "$WEBGPU_DIR"

echo "==> Building test image..."
docker build -f tests/Dockerfile \
    --build-arg BASE_IMAGE="$BASE_IMAGE" -t "$IMAGE" .

echo "==> Running tests..."
docker run --rm \
    -v "$(pwd)/tests/output:/app/tests/output" \
    -v "$(pwd)/tests/baselines:/app/tests/baselines" \
    "$IMAGE"

5. Generate initial baselines#

Run the tests once with UPDATE_BASELINES=1 to create the reference images:

UPDATE_BASELINES=1 ./tests/run_tests.sh

Or pass the variable through Docker:

docker run --rm -e UPDATE_BASELINES=1 \
    -v "$(pwd)/tests/baselines:/app/tests/baselines" \
    my-package-tests

The generated PNGs in tests/baselines/ should be committed to version control. If your repository uses Git LFS for binary files (recommended for PNGs), make sure LFS is set up before committing:

git lfs track "*.png"
git add .gitattributes tests/baselines/
git commit -m "Add baseline images"

GitHub Actions#

The webgpu CI publishes ghcr.io/cerbsim/webgpu-base:latest when its layers change (on pushes to main). Downstream packages can pull this image directly, avoiding the need to check out the webgpu repository or rebuild the base image.

A minimal workflow for a downstream package:

name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  BASE_IMAGE: ghcr.io/cerbsim/webgpu-base

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          lfs: true  # needed if baselines are stored in Git LFS

      - name: Pull base image
        run: docker pull ${{ env.BASE_IMAGE }}:latest

      - name: Build test image
        run: |
          docker build -f tests/Dockerfile \
            --build-arg BASE_IMAGE=${{ env.BASE_IMAGE }}:latest \
            -t my-tests .

      - name: Run tests
        run: |
          docker run --rm \
            -v ${{ github.workspace }}/tests/output:/app/tests/output \
            my-tests

      - name: Upload output on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: test-output
          path: tests/output/

The test image build uses plain docker build (not buildx) so it can see the pulled base image in the local Docker daemon. The test image layer is small (just installing your package + copying tests), so caching it is not necessary.