ENGINEERING VALIDATION · WEBGPU
Porting LivePortrait to Browser WebGPU When 5D GridSample Is Missing
Timeline Studio drives portrait motion with JoyVASA and renders it with LivePortrait. Exporting the models to ONNX was not enough: the combined generator contained rank-5 operations and two 5D GridSample nodes that ONNX Runtime Web could not execute through WebGPU. The solution was a graph rewrite that preserved the model's mathematics, followed by numerical and end-to-end browser validation.
ACCEPTANCE PROTOCOL
What had to be proven
- 01Audit the complete ONNX graph before changing model code or execution providers.
- 02Write the unsupported operator's mathematical behavior before designing a replacement.
- 03Compare fixed-input outputs with maximum, mean and high-percentile absolute error.
- 04Use the matching Asyncify runtime files and serialize large WebGPU session creation.
- 05Reuse stable GPU features across frames while reporting honest render progress.
- 06Validate download, inference, encoding, playback and timeline insertion as one pipeline.
Decompose the avatar pipeline first
The browser workflow is not one model call. JoyVASA converts audio into motion, while LivePortrait performs appearance extraction, motion extraction, lip retargeting, stitching, feature warping and SPADE image generation. Keeping those stages separate creates useful validation boundaries: audio-driver error, keypoint error, warping error and final generator error can be measured independently rather than diagnosed from one incorrect-looking face.
Locate the real WebGPU blocker
The opset-20 combined generator had 277 nodes, including 13 rank-5 convolutions and two rank-5 GridSample nodes. A Linux shared-library plugin from the source repository could not run in the browser, and ONNX Runtime Web's WebGPU execution provider could not directly execute the 5D samplers. Removing the depth dimension, treating it as an arbitrary batch or selecting the nearest slice would all change the function computed by the model.
Rewrite trilinear sampling with supported operations
A target depth coordinate lies between slices d0 and d1. The rewrite performs supported 4D bilinear GridSample on both planes, then interpolates the two results using the fractional depth coordinate alpha. Depth slices are arranged as batches so WebGPU can process them in parallel and then restored to the original spatial layout. A verified shared-XY structure in the sparse-motion grid removes redundant sampling. The rewritten graph grew from 277 to 316 nodes, but every operation could run in the browser.
Prove numerical equivalence
Fixed inputs were passed through both the original 512 by 512 generator and the rewritten graph. The complete output produced a maximum absolute error of 7.8976e-6, a mean absolute error of 1.3853e-7 and a 99.9th-percentile error of 1.7881e-6. JoyVASA's HuBERT and diffusion-denoiser ONNX exports also matched their PyTorch references within 7.75e-6 and 6.44e-6 maximum absolute error. Visual similarity alone would not have provided this evidence.
Solve the browser runtime around the graph
The WebGPU Workers require the matching ONNX Runtime Asyncify JavaScript and WASM pair; the plain or JSEP glue can fail during WebGPU initialization. Large execution-provider sessions are created serially, and the reusable portrait appearance feature remains GPU-resident across frames. The roughly 906 MB avatar bundle stays outside Git, is split into manageable files, loaded from an immutable model revision and cached by the service worker.
Do not confuse success with real time
Timeline Studio offers a mixed-FP16 256-pixel preview tier and a mixed-FP16 512-pixel quality tier. Sparse neural keyframes reduce the number of expensive renders, but validated 512-pixel frames still took roughly 25 to 60 seconds on the test device. The product reports real stages and elapsed time, checks for non-finite or temporally abnormal frames, retries one suspect frame, and falls back to the previous valid frame when necessary.
Finish with an end-to-end media test
A tensor returned by an ONNX session is not a finished product. Acceptance covered model-part downloads, WebGPU initialization, 50-step JoyVASA motion generation, LivePortrait rendering, WebM encoding, browser playback, media-library insertion and visual-track replacement. The resulting 3.56-second WebM showed open and closed mouth states with head and eye motion, and the browser console reported no errors after completion.
REFERENCE
Frequently asked questions
Why not use a custom native plugin in the browser?
Browser runtimes cannot load the original Linux shared library, so the computation must use portable supported operators or a browser-specific implementation.
Is the rewritten GridSample exact?
Floating-point execution introduces small differences, but the measured full-output error remained on the order of 1e-6 to 1e-7.
Is the 512-pixel avatar path real time?
No. It is a quality path with honest progress reporting; sparse neural keyframes and interpolation reduce total work without being presented as real-time inference.
Why pin the model revision?
A fixed revision keeps cached graph files, tensor contracts and runtime assumptions synchronized across sessions.