meat/.agents/plans/gpu.md

# GPU renderer migration direction note

Status: **options open; migration seam agreed.** This note records the current
problem, viable approaches, expected scaling, proposed architecture, migration
order, and decisions still needed. It is not approval to delete the software
renderer or to commit to WebGL2, WebGPU rasterization, or WebGPU compute.

## Goal

Move triangle transformation and pixel rasterization off the CPU so adding more
world geometry and moving actors does not collapse frame rate.

The GPU path must preserve the configurable PS1 look rather than replace it with
a fixed visual preset. `RenderConfig` remains the source of live settings for
internal resolution, vertex snap, texture filtering, lighting, fog, color depth,
dither, and LOD distance.

The current software renderer remains playable during migration. A GPU backend
is added beside it and selected at startup or by a debug option until parity and
performance are proven.

## Decisions already made

- Build a parallel backend instead of replacing the software renderer in place.
- Keep engine world/content boundaries unchanged. GPU work must not reintroduce
  content kinds, registries, or renderer knowledge of frogs, trees, texture names,
  or other game content.
- Keep `RenderScene` as the clone-safe, content-agnostic render projection of a
  live `Level`.
- Keep main-thread frustum culling and scene-local render prototype indexes at
  first. GPU culling is not required to obtain the expected performance gain.
- Optimize for scaling as content grows, not merely for improving the current
  benchmark by a small constant factor.
- Preserve low-resolution rendering and the live PS1 controls.

## Decisions not made

- WebGPU versus WebGL2 as the first GPU backend.
- Fixed-function GPU rasterization versus a custom WebGPU compute rasterizer.
- Whether WebGL2 and WebGPU should both exist long-term.
- Whether the software renderer remains a permanent fallback after a GPU backend
  becomes default.
- Whether GPU output must be pixel-identical to software output or only visually
  equivalent under the same `RenderConfig`.
- Required browser, OS, and device support.
- Whether GPU timing statistics are required for the first usable backend.

## Current baseline

Current final browser benchmark at 384x216:

| Backend | Work median | Work p95 | Frame p95 |
| --- | ---: | ---: | ---: |
| Software, single thread | 30.07 ms | 32.19 ms | 32.60 ms |
| Software, 3 workers | 11.16 ms | 13.35 ms | 16.88 ms |

These numbers are machine- and scene-specific. They are useful as the baseline
for this repository, not as universal predictions.

The software cost grows with transformed triangles, covered pixels, and texture
samples. Workers divide framebuffer rows, but every worker still loops over the
same visible draw groups and transforms the same triangles for its band. More
content therefore keeps increasing CPU work even when screen resolution stays
fixed.

## Remaining implementation options

### Option A: standard WebGPU rasterization

Use WebGPU render pipelines, vertex/index buffers, textures, samplers, a depth
texture, actor instancing, and WGSL shaders. Recreate PS1 traits in shaders and a
post-process pass.

Strengths:

- Uses hardware raster units for the work they are designed to perform.
- Best route to large geometry and actor-count headroom.
- Explicit resource and command model fits immutable compiled `RenderScene` data.
- Storage buffers and compute remain available later without changing API.
- Device-loss handling can fall back to the software backend.

Costs:

- More setup and resource-lifecycle code than WebGL2.
- Uniform/storage alignment and bind-group layout require care.
- Texture upload row pitch must satisfy WebGPU alignment requirements.
- Browser/device support must be checked against actual release targets.
- GPU work timing needs optional timestamp queries or coarse CPU submission timing.

Expected scaling:

Current geometry should normally fit in low single-digit GPU milliseconds on a
reasonable desktop GPU. At this low internal resolution, 50k triangles is small
for fixed-function GPU rasterization. Hundreds of thousands of visible triangles
should remain practical if draw count, state changes, and overdraw stay controlled.
These are estimates, not acceptance results.

### Option B: standard WebGL2 rasterization

Use a WebGL2 context, VAOs, vertex/index buffers, textures, samplers, an offscreen
framebuffer, a depth attachment, GLSL ES 3 shaders, and instanced actor draws.

Strengths:

- Simpler first standard-raster backend.
- Broad and mature browser implementation history.
- All current PS1 effects can be implemented without compute shaders.
- Hardware triangle throughput should be enough for this engine by a wide margin.
- Easier shader/program debugging in many browser tools.

Costs:

- More implicit global state than WebGPU.
- Resource binding and synchronization are less explicit.
- No general compute path if exact software rasterization becomes necessary.
- Large dynamic instance data is less flexible than WebGPU storage buffers.
- A later WebGPU backend would duplicate substantial platform code.

Expected scaling:

For this renderer, standard WebGL2 and standard WebGPU should have similar basic
triangle throughput because both use hardware rasterization. JavaScript draw-call
overhead, material changes, and actor submission are more likely to distinguish
them than raw fill rate at 384x216.

### Option C: WebGPU compute software rasterizer

Port the custom rasterizer to WGSL compute rather than using render pipelines.
This is the path with the best chance of preserving exact custom edge, depth, and
pixel rules, but it does not automatically scale like hardware rasterization.

A serious implementation needs multiple stages:

1. Transform vertices and clip triangles.
2. Compute screen bounds and assign triangles to 8x8 or 16x16 pixel tiles.
3. Store per-tile triangle lists with overflow handling.
4. Rasterize each tile in a workgroup, using workgroup memory where useful.
5. Resolve depth, alpha cutout, texture sampling, lighting, and color.
6. Run color quantization/dither and copy to the canvas.

Expected speed:

A properly tiled compute renderer could plausibly reduce the current scene to
roughly 1-5 ms on a decent desktop GPU. A naive port can be slower than the current
worker renderer. Exact performance depends on tile occupancy, atomics, overdraw,
texture access, and hardware.

Scaling limitations:

- Vertex transformation and binning still grow with triangle count.
- Dense tiles grow with local triangle count and overdraw.
- Triangle-parallel writes contend on depth and color.
- Pixel-parallel loops over every triangle are catastrophically expensive.
- Correct alpha-cutout depth ordering complicates packed atomic updates.
- Reproducing deterministic software ordering can serialize work.

Use this option when exact software-raster behavior is more important than maximum
content headroom. It is not the recommended first answer to the current scaling
problem, but it remains open.

### Option D: CPU rasterizer with GPU presentation

Keep software rendering and upload the completed CPU framebuffer into a GPU texture
for presentation.

Strengths:

- Small migration step.
- Preserves exact current pixels.
- Can establish canvas, texture-upload, scaling, and backend-selection plumbing.
- May simplify or replace Canvas2D presentation.

Limitations:

- Does not move triangle or pixel work off the CPU.
- Does not solve frame-rate collapse as content grows.
- Adds an upload every frame.

This remains useful as an intermediate bridge or diagnostic backend, not as the
destination for the stated performance goal.

### Option E: both WebGPU and WebGL2

Implement a common backend contract, then supply WebGPU, WebGL2, and software
implementations.

Strengths:

- Best runtime coverage while retaining modern WebGPU capabilities.
- Allows direct performance comparison on the same machine and scene.
- Software remains the reference renderer.

Costs:

- Three renderers, shader languages, resource systems, and failure paths.
- PS1 feature fixes must be maintained in multiple implementations.
- Highest test and debugging burden.

Do not start both GPU backends simultaneously unless supported-browser requirements
make that necessary. The common seam should permit a second backend without making
it mandatory.

## Current architecture that should survive

No game-content rewrite is needed.

- `engine/world/Level.ts` owns live actors, collision, terrain, and render projection.
- `engine/render/RenderScene.ts` owns static groups, chunks, billboards, prototypes,
  sky configuration, culling, and scene-local instance data.
- `engine/render/Chunk.ts` owns LOD choice.
- `Mesh` already provides flat position/UV vertices and indexed triangles.
- `Material` already binds a concrete `Texture` and cull setting.
- `ChunkBuilder` already batches static geometry by material object.
- Actor definitions already expose material-bound render prototypes.
- Main-thread frustum culling already prevents invisible chunks and actors from
  reaching the renderer.

GPU compilation should consume those objects directly. Backend-local maps may
deduplicate resources by `Mesh`, `Texture`, `Material`, or `RenderPrototype` object
identity. Such maps are compiled-resource caches, not semantic content registries:
they have no content names, stable IDs, or dispatch behavior.

## Proposed backend seam

Current `app/renderer.ts` exposes software-specific `fb` and `parallel` fields.
Presentation also lives partly in `app/main.ts`. A GPU backend needs a backend-neutral
surface.

Candidate frame input:

```ts
export type RenderFrame = {
  camera: Camera
  viewProjection: Mat4
  visibleChunks: readonly number[]
  instances: readonly RenderInstance[]
  time: number
}
```

Candidate renderer contract:

```ts
export type Renderer = {
  readonly width: number
  readonly height: number
  readonly backend: "software" | "webgl2" | "webgpu"
  reconfigure: (config: RenderConfig) => void
  dispatch: (frame: RenderFrame) => void
  done: () => boolean
  present: () => void
  workMs: () => number | null
  dispose: () => void
}
```

This is a candidate, not a final API. Important properties:

- Main loop no longer reads a software `Framebuffer` to discover dimensions.
- Backend owns presentation. Software calls `putImageData`; WebGL/WebGPU finish a
  pass targeting the canvas.
- Software `done()` polls workers; standard GPU backends can initially report true
  after command submission.
- `workMs()` may be unavailable until GPU timing support exists.
- Renderer owns teardown for workers, buffers, textures, and device/context loss.
- Scene replacement or level reload recompiles backend resources explicitly.

## Module ownership options

Core engine must remain DOM-free and usable by the server.

Recommended initial split:

- `engine/render/RenderScene.ts` keeps backend-neutral scene and frame contracts.
- `app/renderers/SoftwareRenderer.ts` wraps the current worker renderer.
- `app/renderers/WebGpuRenderer.ts` or `app/renderers/WebGlRenderer.ts` owns browser
  API calls and canvas contexts.
- `app/main.ts` selects a backend and uses only the common renderer contract.

If a GPU backend grows too large for `app/`, another viable option is a separate
platform project such as `renderers/webgpu/` that depends on `engine/` but is not
part of DOM-free engine core. Do not add DOM/WebGPU globals to `tsconfig.engine.json`
merely to make placement convenient.

## Standard GPU resource compilation

Compile resources once per `RenderScene` or level load.

### Meshes

- Convert `Mesh.verts` into one `Float32Array` with stride 5: x, y, z, u, v.
- Convert `Mesh.indices` into `Uint16Array` when safe or `Uint32Array` otherwise.
- Create one backend mesh resource per distinct `Mesh` object.
- Preserve separate chunk meshes; merging all chunks would defeat CPU culling.
- Release resources when replacing the scene.

### Textures

- Upload existing packed RGBA bytes through a `Uint8Array` view.
- Keep repeat wrapping.
- Create nearest and linear samplers without mipmaps.
- Handle WebGPU `bytesPerRow` alignment with padded staging rows when needed.
- Deduplicate by `Texture` object identity.

### Materials

- Bind texture resources by `Material.texture` object identity.
- Keep culling as finite renderer capability, not content identity.
- Standard GPU paths will probably need two world pipeline variants: culled and
  double-sided.
- Avoid one shader program or pipeline per game material.

### Prototypes and instances

- Compile each `RenderPrototype` once.
- Group visible instances by prototype each frame.
- Upload transform data once per frame.
- Draw repeated actors with instancing rather than one command per actor.
- Keep prototype indexes local to the compiled scene.

### Billboards

- Replace per-frame CPU quad construction with one static unit quad.
- Supply position/size as instance data.
- Derive camera-right direction in the vertex shader so billboards stay upright.

## Standard GPU frame structure

### Pass 1: scene color and depth

Render at `RenderConfig.internalWidth` x `RenderConfig.internalHeight` into an
offscreen RGBA color texture plus depth attachment.

Suggested order:

1. Draw full-screen sky with depth writes disabled.
2. Draw static groups.
3. Draw visible chunks using CPU-selected near/far groups.
4. Draw billboards with alpha cutout and no culling.
5. Draw actor prototype instances.

### Pass 2: quantization and presentation

Sample the offscreen color texture and apply color-depth quantization plus Bayer
dither in a full-screen pass. Write to the canvas texture. Keep CSS integer scaling
and `image-rendering` behavior where useful.

This two-pass shape matches current behavior better than quantizing each material
fragment independently because software quantization runs after the complete frame,
including sky.

## Mapping current raster features to shaders

| Current feature | Standard GPU implementation | Parity risk |
| --- | --- | --- |
| Perspective-correct UV | Native interpolation | Low; this is already desired behavior |
| Internal low resolution | Offscreen render target at configured size | Low |
| Nearest/linear texture filter | Select sampler from `RenderConfig` | Low |
| No mipmaps | Allocate only base level and use non-mipmap sampler | Low |
| Vertex snap | Snap projected screen coordinates in vertex shader, then rebuild clip xy | Medium; edge rules differ |
| Flat lighting | Fragment derivatives of world/view position, or baked face normals | Medium |
| Distance fog | Fragment shader using view-space distance | Low |
| Alpha cutout | Fragment `discard` below alpha threshold | Low |
| Backface culling | Culled and double-sided pipeline variants | Low; front-face sign must be checked |
| Color depth | Full-screen post-process quantization | Low |
| Bayer dither | Full-screen integer pixel-coordinate lookup | Low |
| 1/w depth behavior | Native perspective depth or explicit fragment depth | Medium; exact values differ |
| Near clipping | Native clipping | Low visually, not pixel-identical |
| Procedural sky | Full-screen sky shader port | Medium; noise parity must be tested |
| Chunk frustum culling | Keep existing CPU `RenderScene.visibleChunks` | Low |
| Actor culling | Keep existing `Level.visibleInstances` | Low |
| Chunk LOD | Keep existing CPU `Chunk.isFar` selection | Low |

Vertex snap sketch:

```text
clip = projection * view * model * position
ndc = clip.xy / clip.w
pixel = (ndc * 0.5 + 0.5) * internalResolution
pixel = round(pixel / snap) * snap
clip.xy = ((pixel / internalResolution) * 2.0 - 1.0) * clip.w
```

Y orientation and half-pixel conventions differ between APIs and need screenshot
tests. Do not guess front-face or snap signs from the software renderer.

## Flat-lighting options

### Fragment derivatives

Calculate a face normal from `cross(dpdx(position), dpdy(position))` in the fragment
shader. This preserves flat faceted lighting without changing mesh data.

Pros:

- Smallest geometry change.
- Naturally one normal per rasterized face.
- Available in WebGL2 fragment shaders and WGSL fragment shaders.

Cons:

- Exact shade may differ from the CPU cross-product calculation.
- Orientation and two-sided handling need validation.

### Baked face normals

Duplicate shared vertices per triangle and store one face normal per vertex.

Pros:

- Explicit and predictable.
- Can reproduce CPU directional-light math closely.

Cons:

- Increases vertex memory.
- Requires mesh compilation to expand indexed geometry.
- Changes cache behavior.

Both options remain open. Derivatives are the recommended first implementation.

## WebGPU-specific notes

- Request adapter/device once and handle failed acquisition cleanly.
- Configure the canvas using the preferred canvas format.
- Use an offscreen `rgba8unorm` scene texture and `depth24plus` initially.
- Keep frame uniforms in a uniform buffer.
- Use dynamic uniform offsets, instance vertex attributes, or storage buffers for
  model transforms. Do not repeatedly overwrite one model uniform before submit.
- Respect 256-byte dynamic-uniform alignment.
- Group actors by prototype and use `instance_index` for transform lookup.
- Keep bind-group layouts stable across materials.
- Separate frame data, texture views, and samplers so changing texture filtering
  does not force rebuilding every texture resource.
- Listen for device loss and switch to software rather than leaving a dead canvas.
- Treat timestamp queries as optional capability.
- Recreate size-dependent color/depth textures after `RenderConfig` resolution
  changes.

## WebGL2-specific notes

- Request `webgl2` with alpha and antialias settings chosen explicitly.
- Use one VAO per compiled mesh.
- Use an FBO with RGBA color texture and depth renderbuffer/texture.
- Use GLSL ES 3 vertex/fragment programs for world, billboard, sky, and post passes.
- Use `drawElementsInstanced` and per-instance matrix attributes for actors.
- Use `dFdx`/`dFdy` fragment derivatives for flat lighting if selected.
- Use `EXT_disjoint_timer_query_webgl2` only when available.
- Handle context loss/restoration by releasing and recompiling scene resources.
- Reset or centralize state changes; implicit stale state is a major WebGL failure
  mode.

## WebGPU compute-raster details

Avoid these naive designs:

- One compute invocation per pixel looping over every triangle.
- One invocation per triangle writing non-atomic color and depth.
- A global atomic lock per pixel around full shading.
- Unbounded fixed-size tile lists with silent overflow.
- Recreating CPU row-band splitting on the GPU.

Open design questions for compute:

- Tile size and maximum triangle references per tile.
- Multi-pass prefix sums versus fixed tile capacity.
- Depth encoding suitable for atomic comparison.
- How alpha-cutout fragments update depth and color consistently.
- Whether exact triangle order is required for equal-depth fragments.
- Whether perspective-correct UV and texture filtering use native sampling.
- How clipping-generated triangles enter tile lists.
- Whether post-process quantization remains a separate render pass.

Do not implement compute first unless pixel parity is declared a hard requirement
after testing a standard raster prototype.

## Draw-call and batching strategy

Current chunks are already grouped by material, which is suitable for GPU upload.
Culling should remain more important than globally merging geometry.

Initial strategy:

- Draw each visible chunk group separately.
- Skip invisible chunks on CPU.
- Select near/far groups on CPU.
- Sort or bucket visible groups by cull pipeline and material only if profiling
  shows draw-call overhead matters.
- Instance actors by prototype.
- Use one unit-quad mesh for billboards.

Possible later optimizations:

- Merge neighboring chunk groups only if visibility remains acceptably granular.
- Multi-draw or indirect draws where API support and complexity justify them.
- GPU frustum/occlusion culling only after CPU submission becomes measured cost.
- Texture atlases or arrays only after material binding becomes measured cost.

Do not add batching abstractions before a profile identifies the bottleneck.

## Performance expectations and growth testing

Standard GPU rasterization is recommended for the content-scaling goal because
hardware raster units remove the CPU per-triangle/per-pixel loop. However, GPU
rendering can still lose frame rate through excessive draws, overdraw, huge texture
uploads, per-frame allocation, or synchronization.

Benchmark growth levels, not only the current scene:

| Scenario | Purpose |
| --- | --- |
| 1x current content | Parity and baseline |
| 2x static props | Early scaling slope |
| 5x static props | Dense-world target |
| 10x static props | Stress and culling behavior |
| 2x/5x moving actors | Instance upload and draw scaling |
| Worst forest camera | Fill, overdraw, and visible draw count |
| Empty clearing | Fixed per-frame overhead |

Record:

- CPU simulation time.
- CPU render submission time.
- GPU time when available.
- Median, p95, and max frame time.
- Visible triangles, draw count, and visible actor count.
- GPU memory and scene compilation time where practical.

Candidate budget, not yet agreed:

- Keep renderer p95 below roughly 10 ms at the chosen content target, leaving
  room inside a 16.67 ms frame for simulation, input, browser, and presentation.
- Demonstrate a substantially flatter frame-time slope than software at 2x and 5x
  content.

## Migration sequence

### Phase 1: isolate backend contract

1. Introduce `RenderFrame` and backend-neutral renderer dimensions/lifecycle.
2. Wrap current software renderer without changing output.
3. Move Canvas2D presentation behind the software backend.
4. Keep `?bench=st|mt` working.
5. Add explicit backend query selection such as `?renderer=software|webgl2|webgpu`.

Acceptance:

- Software screenshots and benchmark remain unchanged.
- Preset switching, resize, worker fallback, and presentation still work.

### Phase 2: GPU scene compiler

1. Traverse `RenderScene` once.
2. Deduplicate meshes, textures, materials, and prototypes by object identity.
3. Upload immutable resources.
4. Record compile time and resource counts.
5. Add disposal and level-reload behavior.

Acceptance:

- Compiling a scene does not mutate engine scene data.
- No content names or registries enter backend code.
- Repeated object references produce one GPU resource.

### Phase 3: minimum standard-raster world

1. Render static groups and visible chunk groups.
2. Add depth testing and material culling.
3. Add texture sampling.
4. Keep CPU culling and LOD.
5. Use a simple clear color before sky parity exists.

Acceptance:

- Geometry, UVs, winding, depth, and near/far LOD are visibly correct.
- Current scene already shows major software-to-GPU work-time reduction.

### Phase 4: actors and billboards

1. Compile render prototypes.
2. Group visible actors by prototype.
3. Upload per-frame transforms.
4. Draw actors with instancing.
5. Draw billboards from a shared unit quad.
6. Add alpha cutout.

Acceptance:

- No actor content dispatch exists in backend.
- Worker scene-local prototype semantics remain backend-neutral.
- Dynamic actor count scaling is measured.

### Phase 5: PS1 look parity

1. Implement vertex snap.
2. Implement nearest/linear sampler selection.
3. Implement flat lighting.
4. Implement fog.
5. Implement color-depth quantization and Bayer dither in post-processing.
6. Validate `standard`, `soft`, `clean`, and `ps1` presets.

Acceptance:

- Every current `RenderConfig` knob works live.
- No PS1 trait is silently hardcoded into GPU shaders.

### Phase 6: sky parity

1. Port gradient and sun.
2. Port skybox sampling.
3. Port basic cumulus.
4. Port fancy cumulus.
5. Preserve reduced cloud sampling where still useful.

Acceptance:

- Sky animation and panorama orientation match visually.
- Cloud cost is profiled separately at each internal resolution.

### Phase 7: reliability and selection

1. Handle WebGPU device loss or WebGL context loss.
2. Fall back to software when GPU initialization fails.
3. Verify preset switching and resize during active rendering.
4. Add screenshot comparison and growth benchmarks.
5. Decide default backend only from support and measured results.

## Validation plan

### Visual scenes

Capture stable views that expose different failure modes:

- Room edge crossing near plane.
- Terrain horizon with fog.
- Dense trees and boulders at LOD transition.
- Flowers and sprite alpha cutout.
- Frog/bee/robin close and far.
- Skybox poles and seam.
- Basic and fancy clouds.
- Backface-culling winding checks.

### Comparison policy

If standard GPU rasterization is selected, require visual equivalence rather than
bit-identical frames unless exact parity is later declared. Native clipping, depth,
edge fill, derivatives, and floating-point behavior will differ.

If compute rasterization is selected specifically for exactness, define which
parts must match bit-for-bit before implementation. "Exact" cannot remain vague.

### Automated checks

- GPU scene compiler imports no game modules.
- Renderer backends import no concrete game modules.
- GPU compilation creates no stable content IDs.
- Software tests continue passing.
- Shader compilation errors fail backend initialization clearly.
- Scene replacement releases old resources.
- Config switch and resize recreate only size/config-dependent resources.
- Device/context loss reaches a working fallback.

## Backend choice guide

Choose standard WebGPU first when:

- Target browsers/devices support WebGPU adequately.
- Future compute or storage-buffer work matters.
- Explicit resource management is worth the larger initial implementation.
- Software fallback is acceptable where WebGPU is absent.

Choose WebGL2 first when:

- Broad browser/device coverage is the priority.
- Fastest path to hardware rasterization is the priority.
- Current feature set is enough and compute is not required.
- Maintaining software fallback covers correctness/reference needs.

Choose WebGPU compute first only when:

- Exact custom raster behavior is a hard product requirement.
- Standard GPU raster tests prove shader emulation insufficient.
- The project accepts a tile-binning/atomic rasterizer as a major subsystem.
- WebGPU-only support is acceptable.

Choose both standard GPU APIs only when:

- Browser support requirements cannot be met by one GPU API plus software fallback.
- The maintenance cost is explicitly accepted.

## Current recommendation

Start with the backend seam, then prototype **standard WebGPU rasterization** while
keeping software fully operational. Standard WebGL2 remains a valid first backend
if target-browser research favors it. Measure one textured static world pass before
porting every visual feature.

Do not begin with compute unless exact edge/depth behavior is declared mandatory.
For the stated problem, standard hardware rasterization offers the strongest chance
of adding substantially more content without frame time growing like the current
CPU rasterizer.

## Questions required before implementation

1. Which browsers, OS versions, and device classes must run the GPU path?
2. Is software-renderer pixel identity required, or is visual equivalence enough?
3. What content-growth target must hold 60 fps: 2x, 5x, 10x, or a concrete level?
4. Is the software backend a permanent supported mode or only migration/reference?
5. Is GPU timing required in the HUD for the first version?
6. Should backend selection be automatic, query-driven, or user-configurable?
7. Is WebGPU compute still desired after a standard-raster visual prototype exists?

Until these are answered, keep all options above open and avoid API choices that
make WebGL2, WebGPU rasterization, or WebGPU compute unnecessarily impossible.