ENGINEERING VALIDATION · WEBGPU
Porting LivePortrait to Browser WebGPU When 5D GridSample Is Missing
Timeline Studio drives portrait motion with JoyVASA and renders it with LivePortrait. Exporting the models to ONNX was not enough: the combined generator contained rank-5 operations and two 5D GridSample nodes that ONNX Runtime Web could not execute through WebGPU. The solution was a graph rewrite that preserved the model's mathematics, followed by numerical and end-to-end browser validation.
ACCEPTANCE PROTOCOL
What had to be proven
- 01Audit the complete ONNX graph before changing model code or execution providers.
- 02Write the unsupported operator's mathematical behavior before designing a replacement.
- 03Compare fixed-input outputs with maximum, mean and high-percentile absolute error.
- 04Use the matching Asyncify runtime files and serialize large WebGPU session creation.
- 05Reuse stable GPU features across frames while reporting honest render progress.
- 06Validate download, inference, encoding, playback and timeline insertion as one pipeline.
Decompose the avatar pipeline first
The browser workflow is not one model call. JoyVASA converts audio into motion, while LivePortrait performs appearance extraction, motion extraction, lip retargeting, stitching, feature warping and SPADE image generation. Keeping those stages separate creates useful validation boundaries: audio-driver error, keypoint error, warping error and final generator error can be measured independently rather than diagnosed from one incorrect-looking face.
Locate the real WebGPU blocker
The opset-20 combined generator had 277 nodes, including 13 rank-5 convolutions and two rank-5 GridSample nodes. A Linux shared-library plugin from the source repository could not run in the browser, and ONNX Runtime Web's WebGPU execution provider could not directly execute the 5D samplers. Removing the depth dimension, treating it as an arbitrary batch or selecting the nearest slice would all change the function computed by the model.
Rewrite trilinear sampling with supported operations
A target depth coordinate lies between slices d0 and d1. The rewrite performs supported 4D bilinear GridSample on both planes, then interpolates the two results using the fractional depth coordinate alpha. Depth slices are arranged as batches so WebGPU can process them in parallel and then restored to the original spatial layout. A verified shared-XY structure in the sparse-motion grid removes redundant sampling. The rewritten graph grew from 277 to 316 nodes, but every operation could run in the browser.
Prove numerical equivalence
Fixed inputs were passed through both the original 512 by 512 generator and the rewritten graph. The complete output produced a maximum absolute error of 7.8976e-6, a mean absolute error of 1.3853e-7 and a 99.9th-percentile error of 1.7881e-6. JoyVASA's HuBERT and diffusion-denoiser ONNX exports also matched their PyTorch references within 7.75e-6 and 6.44e-6 maximum absolute error. Visual similarity alone would not have provided this evidence.
Solve the browser runtime around the graph
The WebGPU Workers require the matching ONNX Runtime Asyncify JavaScript and WASM pair; the plain or JSEP glue can fail during WebGPU initialization. Large execution-provider sessions are created serially, and the reusable portrait appearance feature remains GPU-resident across frames. The roughly 906 MB avatar bundle stays outside Git, is split into manageable files, loaded from an immutable model revision and cached by the service worker.
Do not confuse success with real time
Timeline Studio offers a mixed-FP16 256-pixel preview tier and a mixed-FP16 512-pixel quality tier. Sparse neural keyframes reduce the number of expensive renders, but validated 512-pixel frames still took roughly 25 to 60 seconds on the test device. The product reports real stages and elapsed time, checks for non-finite or temporally abnormal frames, retries one suspect frame, and falls back to the previous valid frame when necessary.
Finish with an end-to-end media test
A tensor returned by an ONNX session is not a finished product. Acceptance covered model-part downloads, WebGPU initialization, 50-step JoyVASA motion generation, LivePortrait rendering, WebM encoding, browser playback, media-library insertion and visual-track replacement. The resulting 3.56-second WebM showed open and closed mouth states with head and eye motion, and the browser console reported no errors after completion.
Make tensor contracts executable documentation
Graph surgery becomes fragile when shapes are remembered only in a notebook. Every stage therefore needs a contract that records tensor name, rank, axis meaning, numeric type, expected range and ownership. The five-dimensional feature volume is not simply a large image: batch, channel, depth, height and width each carry different semantics, while the motion grid describes normalized sampling coordinates. The rewrite keeps these meanings visible through named reshape and transpose steps and validates intermediate shapes before the final generator runs. Fixed fixtures catch accidental axis swaps that might still produce a plausible portrait. Contract tests also cover dynamic dimensions, mixed-precision boundaries and the point at which GPU data must return to CPU memory for encoding. This documentation makes later runtime upgrades safer because a changed operator, execution provider or model revision can be checked against a precise interface instead of judged only by whether one example appears to work.
Validate temporal behavior, not just individual frames
A portrait pipeline can pass pixel-level tests and still fail as video. Small independent errors may flicker, head pose can jump at sparse keyframes, lip motion may drift from audio, and a single non-finite frame can poison an encoder. Timeline Studio therefore treats time as another validation dimension. It compares adjacent motion and rendered frames, checks that mouth movement appears across voiced regions, measures whether interpolation preserves stable identity features and retries an abnormal neural keyframe once before falling back to the previous valid image. Playback is inspected after encoding because timestamp rounding and frame pacing can introduce defects that tensors do not reveal. The final acceptance clip includes still and moving regions, open and closed mouth states, eye and head motion, and timeline replacement. That broader test answers the product question: not merely whether WebGPU produced numbers, but whether a user received a playable, synchronized and recoverable media asset.
REFERENCE
Frequently asked questions
Why not use a custom native plugin in the browser?
Browser runtimes cannot load the original Linux shared library, so the computation must use portable supported operators or a browser-specific implementation.
Is the rewritten GridSample exact?
Floating-point execution introduces small differences, but the measured full-output error remained on the order of 1e-6 to 1e-7.
Is the 512-pixel avatar path real time?
No. It is a quality path with honest progress reporting; sparse neural keyframes and interpolation reduce total work without being presented as real-time inference.
Why pin the model revision?
A fixed revision keeps cached graph files, tensor contracts and runtime assumptions synchronized across sessions.