TEMPORAL SUPER RESOLUTION · WEBGPU
Four Times the Frame: How NanoVSR Restores Video in the Browser
A 4× badge is easy to put on an upscaler. The difficult part is deciding what four times actually means when the source is a trimmed video, when five neighboring frames contain different evidence, when the neural network accepts only 320×180 input, and when an already-large source should not be reduced merely to satisfy a model. Timeline Studio's new NanoVSR path treats those questions as engineering constraints. It runs separate one-frame and five-frame FP16 ONNX models through WebGPU, restores low-resolution images or video locally, protects source resolution when the model's effective output would be smaller, and recomposes video at 12 fps with the available source audio. The result is a new editable asset, not a claim that invented texture is original detail.
CHECKLIST
Practical review checklist
- 01Use genuinely low-resolution or visibly compressed material; clean high-resolution footage has less to gain.
- 02Judge faces, lettering, rails, fences, foliage and other repeated fine structure for invented or unstable texture.
- 03Remember that the current model canvas is 320×180 and the nominal neural output is 1280×720 before aspect-ratio cropping.
- 04Review motion at normal speed as well as paused frames because temporal artifacts can hide in still comparisons.
- 05Check that trimmed source timing and the optional audio track remain aligned in the composed MP4.
- 06Keep clips short on memory-constrained devices: decoded bitmaps, FP16 inference, PNG frames and FFmpeg.wasm all consume resources.
Super resolution is reconstruction, not a larger canvas
Ordinary image scaling can create four times as many pixels with a deterministic filter. It cannot decide whether a soft diagonal should become a clean rail, whether block noise is texture, or whether two blurred edges belonged to one thin line. NanoVSR is trained to reconstruct a higher-resolution frame from low-resolution evidence. That makes it useful for old web video, compressed screen captures and small source clips, but it also changes the promise. The model can generate plausible high-frequency structure; it cannot recover information that was never recorded. Timeline Studio therefore calls the operation restoration, shows a before-and-after divider and keeps the original media available. A sharper result is an interpretation of the source, not forensic proof of what the scene originally contained.
Two models serve two kinds of evidence
The Worker selects between two pinned FP16 ONNX graphs. The image path uses NanoVSR 644K T1 and accepts one frame. The video path uses the T5 graph and accepts a temporal window of five frames. Both run through ONNX Runtime Web with the WebGPU execution provider and full graph optimization. The distinction matters: a still image has only spatial neighbors, while video can use adjacent moments to reinforce an edge or texture that is weak in one frame. The implementation does not pass a single frame through the image model repeatedly and label the sequence video restoration. It gives the temporal model five consecutive bitmaps so motion and repeated evidence are part of the inference input.
The model's working canvas is always 320 by 180
NanoVSR does not receive the source at arbitrary dimensions. Each bitmap is fitted inside a 320×180 canvas using contain geometry. Width and height are rounded to even values, the unused area is filled with black and the resulting pixels are converted into planar float RGB values between zero and one. Aspect ratio is preserved, so portrait or non-16:9 media receives letterbox space rather than being stretched. The tensor shape is 1×T×3×180×320, where T is one for images or five for video. A fixed model canvas makes memory and model shapes predictable, but it also defines the ceiling of neural evidence: the network reasons from at most 57,600 input pixels per frame.
Four times means a 1280 by 720 neural canvas
The model output is four times the working width and height: 1280×720. Timeline Studio converts the planar output back to RGBA, then crops away the scaled letterbox area using the same contain rectangle. A 16:9 source can therefore produce the full 1280×720 result, while other aspect ratios produce the corresponding four-times crop. For a genuinely small source, that is a real fourfold enlargement of the model-space image. The phrase should not be misread as unlimited multiplication of any input. A 1920×1080 upload does not emerge at 7680×4320 from this implementation. The useful statement is precise: NanoVSR performs 4× reconstruction from its fixed 320×180 working representation.
Large sources are protected from an accidental downgrade
A fixed neural output creates an awkward case. If the source already contains more pixels than the cropped 4× model result, replacing it with that result would call itself enhancement while reducing resolution. The Worker detects this by comparing source area with model-output area. When the source is larger, it renders the neural crop back at the original width and height instead of shrinking the asset. It then blends 88 percent source color, 12 percent model output and a small protected-detail term derived from the source's local difference from a five-sample blur. This path keeps the original dimensions and most source information. It is conservative enhancement, not a second claim of 4× output.
Five-frame groups make video restoration temporal
Timeline Studio samples the retained source range at 12 frames per second and processes the frames in groups of five. For each group it seeks the video element to sourceStart plus the frame timestamp, creates five ImageBitmaps and transfers them to the Worker. Near the end, when fewer than five real frames remain, the Worker repeats the last available bitmap to fill the temporal tensor while returning only the requested number of outputs. The T5 model produces a corresponding restored frame for each requested position. Groups are non-overlapping in the current implementation. That keeps the control flow and memory bounded, though it also means an editor should pay special attention to visual continuity at five-frame group boundaries.
Twelve frames per second is an explicit product trade-off
The public dialog states that video is restored at 12 fps, and the restoration hook passes that value into the processor. The library clamps requested rate between one and fifteen frames per second. Twelve lowers the number of WebGPU inferences, decoded frames, PNG files and encoder inputs compared with preserving a 24, 30 or 60 fps source. It can be appropriate for short demonstrations, archival material or intentionally restrained motion, but it does not preserve the temporal smoothness of every source. The output duration remains tied to the retained source range, while fewer unique frames describe that duration. This is why full-speed playback matters: a crisp paused frame cannot reveal whether motion cadence is acceptable.
Trimmed clips begin at their actual source range
The processor reads sourceStart and sourceDuration from the selected segment instead of assuming the original file begins at zero. It limits duration to the media still available after that offset, calculates total frames from retained source duration and frame rate, then seeks every sample relative to sourceStart. This is the same non-destructive time discipline required elsewhere in a timeline editor. If a user trimmed ten seconds from the beginning before restoration, NanoVSR should enhance the visible clip, not silently spend minutes processing discarded footage. The audio composer receives the same source start and duration so picture and sound are cut from the same portion of the asset.
Model delivery is local-first, not model-free
Inference happens in a Worker on the user's GPU, but the application still needs model weights. Timeline Studio requests a pinned revision of either the T1 or T5 file from its model repository, choosing the preferred Hugging Face or ModelScope mirror according to settings and language. Download progress is exposed, and the ArrayBuffer is saved in a dedicated Cache Storage entry. Separate session promises are kept for image and video modes because their graph shapes differ. Later runs can report that the model came from cache before initializing WebGPU. Local processing therefore means source frames stay on the device; it does not mean the browser performs restoration without downloading software and weights.
Cancellation reaches the Worker and the encoder
Long restoration jobs need a real stop path. The hook owns an AbortController. Aborting a NanoVSR request posts a cancel message to the Worker and rejects the pending promise with an AbortError. The Worker records canceled request IDs, closes transferred bitmaps and suppresses progress or result messages for that request. During video composition, the same signal is passed into frame production and FFmpeg.wasm. If cancellation reaches encoding, the application terminates the FFmpeg instance and clears its cached loader. The original visual segment remains in place until a completed preview is explicitly applied. This is safer than swapping a half-produced asset into the timeline and hoping a later retry repairs it.
PNG frames become an H.264 MP4 with optional audio
Once each restored video frame is available as PNG, the shared browser compositor writes the sequence into FFmpeg.wasm's virtual filesystem. It encodes H.264 with the veryfast preset, CRF 18, yuv420p and faststart. When the selected source can be resolved to a Blob, it is added as an audio input, trimmed with the same source offset and duration, mapped as optional audio and encoded to AAC at 192 kbps. The shortest flag prevents one stream from extending beyond the other. The output is an MP4 Blob, and temporary frames, audio and output files are removed. This is a real video deliverable, not merely a high-resolution frame shown over the old clip.
The restored clip remains an editable project asset
Finishing inference does not immediately replace the current clip. The dialog presents synchronized source and restored playback, a timeline scrubber and a draggable comparison line. Apply creates a new user asset named with a nanovsr-4x suffix, records the output dimensions and duration, and swaps the selected segment only after approval. Enhancement metadata stores both original and processed source fields, including Blobs, dimensions and source timing, plus backend, frame rate and total frame count. If the same enhanced segment is reopened, the hook can recover the processed preview while retaining a path back to the original. Restoration becomes a reversible project decision rather than a destructive file conversion.
What the 644K model can and cannot improve
A compact model and fixed input are practical for browser GPU execution, but they do not make every source production-master quality. NanoVSR can strengthen edges, reduce the appearance of block compression and create more coherent texture from low-resolution evidence. It may also oversharpen, invent repeated patterns, alter small letters, simplify faces or produce unstable detail around fast movement and occlusion. Five-frame input gives temporal context, not a guarantee of perfect temporal consistency. A 12 fps output may be too sparse for sports, fast camera moves or high-frame-rate UI recordings. WebGPU support, adapter limits, cache space and FFmpeg memory remain device constraints. The right workflow starts with a representative short clip and accepts the result only after normal-speed review.
The useful claim is narrower—and better
Calling the feature 4× is correct at the neural model boundary: 320×180 becomes 1280×720 before aspect-ratio cropping. Product communication becomes misleading only when it implies that every source dimension is multiplied by four or that generated detail is recovered truth. Timeline Studio's code supports a more defensible claim. It uses a one-frame model for images and a five-frame temporal model for video, protects higher-resolution sources, preserves the selected source range and available audio, exposes progress and cancellation, and returns a reviewable editable asset. That narrower explanation gives editors enough information to decide when browser-local super resolution is genuinely useful—and when the original footage is already the better source.
REFERENCE
Frequently asked questions
Does every video become four times wider and taller?
No. The neural working canvas scales from 320×180 to 1280×720 before aspect-ratio cropping. If the source is already larger than the effective model output, Timeline Studio preserves the source dimensions and blends protected source detail instead of shrinking it.
Does NanoVSR run locally?
Inference runs in a browser Worker through WebGPU. The pinned ONNX model must first download from a configured mirror and is then stored in Cache Storage.
Why does video use five frames at once?
The T5 model uses temporal context from five consecutive frames, giving it evidence that a one-frame image model cannot use.
What frame rate does the restored video use?
The current Timeline Studio workflow samples and composes restored video at 12 fps. The library accepts a clamped range from 1 to 15 fps.
Is the source audio preserved?
When the source can be resolved to a Blob with audio, FFmpeg.wasm trims the same source range and adds the optional audio stream to the H.264 MP4 as AAC.
Can it restore exact lost detail?
No. Super resolution reconstructs plausible high-frequency detail from low-resolution evidence. Generated texture should not be treated as forensic recovery.
Does applying restoration overwrite the original clip?
It creates a new PNG or MP4 asset and records the original and processed sources in enhancement metadata, so the editable project retains the original media path.