Four Times the Frame: How NanoVSR Restores Video in the Browser
A 4× badge is easy to put on an upscaler. The difficult part is deciding what four times actually means when the source is a trimmed video, when five neighboring frames contain different evidence, when the neural network accepts only 320×180 input, and when an already-large source should not be reduced merely to satisfy a model. Timeline Studio's new NanoVSR path treats those questions as engineering constraints. It runs separate one-frame and five-frame FP16 ONNX models through WebGPU, restores low-resolution images or video locally, protects source resolution when the model's effective output would be smaller, and recomposes video at 12 fps with the available source audio. The result is a new editable asset, not a claim that invented texture is original detail.
A three-second pan, its actual export and the wait behind it
We ran a controlled coffee-photo pan through Timeline Studio's existing browser restoration library and WebGPU worker. The 320 × 180 input produced a playable 1280 × 720 H.264 file with an AAC audio track. The run took 50.0 seconds with model files already prepared locally. That establishes a completed short-sample pipeline; the visual comparison below lets you decide whether the change is useful.
Left: decoded input enlarged with Lanczos. Right: the actual NanoVSR export. Both halves use the same display dimensions and 12-fps comparison clock. The sound is a synthetic 440 Hz tone. This comparison is a separately encoded viewing aid; untouched files are linked below.
Follow the white cup rim, the small bubbles and lines in the foam, and the spoon as the image moves. Judge texture at the same size; the number of output pixels alone cannot establish recovered detail. Then play the original 24-fps file and the 12-fps result separately. Halving the number of sampled moments changes motion cadence, even when the duration and sound are retained.
The input is an artificial camera pan made from one photograph. It has no independently moving objects, changing expression, occlusion or genuine camera shake. In the one-second comparison frame, the model makes the cup rim and spoon edges more pronounced; the finer surface texture also changes. Those changes are visible, but the sample cannot tell us which new detail corresponds to the original photographed scene. This is a controlled texture-and-motion example, not evidence for faces, sport, natural handheld footage or every five-frame boundary. There is no high-resolution captured-video reference, so we do not report PSNR, SSIM or a percentage quality improvement.
Time spent before and during encoding
One local run, measured at the calling browser page
Interval
Recorded time
Includes
Start to encoder preparation
39.7 s
Read/seek input, worker and model initialization, restoration and PNG creation.
Encoder preparation
0.6 s
Load the browser encoder and stage input frames/audio.
Encoding to completed result
9.7 s
Browser FFmpeg encoding and result creation.
Complete call
50.0 s
Ends when the result Blob is returned; excludes saving the artifact to disk.
The environment was the Codex in-app Chromium browser on macOS, reporting an Apple Metal-3 WebGPU adapter and Chrome 152 in its user agent. The browser did not expose a precise GPU model. We started a fresh worker for this recorded call, but this browser and GPU had already run a preparation trial. The two pinned model files were read from existing local disk files, hash-checked and placed in the worker's normal named Cache Storage. Internet model-transfer time is excluded, and driver/shader caches were not reset. These conditions are part of the timing result.
This sample took about 16.7 times its playback duration to process. That ratio describes this call only. It is not a throughput forecast for longer media, a comparison between devices, or a claim that every first visit takes the same time. Begin with a representative short clip and decide whether the visual result justifies another run.
The last frame exposed a model-selection detail
Thirty-six output frames create seven complete five-frame groups and one final singleton. The reviewed worker selects its T1 image model when it receives one bitmap and one requested output; it selects T5 for other groups. The run log therefore shows a second cached-model/compile stage at frame 36. A tail of two to four frames is padded for T5, while this one-frame tail takes T1. The earlier description that every short tail always repeats frames into T5 was too broad.
The calling library was used unchanged at its default 12 fps. We did not change the model-selection behavior for this demonstration. Keep the last frame and group transitions in your playback review rather than treating the entire result as one uniform five-frame inference.
Repeat the sample and check the actual media
Download the input and original browser output above. Probe records contain codec, frame-rate, dimensions, sample-aspect ratio, duration and frame counts. Both files decoded completely with FFmpeg without an error.
Use the local reproduction harness with the specified Timeline Studio source and existing dependencies/models. It invokes the same library outside the editor's interface. Preserve your own environment, input, output and logs when comparing.
This test exercised the local production restoration library, WebGPU worker and browser encoder. It did not click through the hosted editor, apply the asset to a timeline, verify cancellation or test mobile hardware. The comparison video was composed afterward with native FFmpeg and is labelled separately from the untouched browser export.
Coffee photograph by Rachel Michetti, courtesy of Pikolo Espresso Bar; CC0 as documented by scikit-image. Pan, tone, restoration run and comparison prepared for this article. NanoVSR models are MIT-licensed; see the model notice and method and limitations.
01
Super resolution is reconstruction, not a larger canvas
Ordinary image scaling can create four times as many pixels with a deterministic filter. It cannot decide whether a soft diagonal should become a clean rail, whether block noise is texture, or whether two blurred edges belonged to one thin line. NanoVSR is trained to reconstruct a higher-resolution frame from low-resolution evidence. That makes it useful for old web video, compressed screen captures and small source clips, but it also changes the promise. The model can generate plausible high-frequency structure; it cannot recover information that was never recorded. Timeline Studio therefore calls the operation restoration, shows a before-and-after divider and keeps the original media available. A sharper result is an interpretation of the source, not forensic proof of what the scene originally contained.
02
Two models serve two kinds of evidence
The Worker selects between two pinned FP16 ONNX graphs. The image path uses NanoVSR 644K T1 and accepts one frame. Video groups normally use the T5 graph with five-frame input; a final singleton selects T1, as documented in the recorded sample. Both run through ONNX Runtime Web with the WebGPU execution provider and full graph optimization. The distinction matters: a still image has only spatial neighbors, while video can use adjacent moments to reinforce an edge or texture that is weak in one frame. The implementation does not pass a single frame through the image model repeatedly and label the sequence video restoration. It gives the temporal model five consecutive bitmaps so motion and repeated evidence are part of the inference input.
03
The model's working canvas is always 320 by 180
NanoVSR does not receive the source at arbitrary dimensions. Each bitmap is fitted inside a 320×180 canvas using contain geometry. Width and height are rounded to even values, the unused area is filled with black and the resulting pixels are converted into planar float RGB values between zero and one. Aspect ratio is preserved, so portrait or non-16:9 media receives letterbox space rather than being stretched. The tensor shape is 1×T×3×180×320, where T is one for images and singleton video tails, or five for other video groups. A fixed model canvas makes memory and model shapes predictable, but it also defines the ceiling of neural evidence: the network reasons from at most 57,600 input pixels per frame.
04
Four times means a 1280 by 720 neural canvas
The model output is four times the working width and height: 1280×720. Timeline Studio converts the planar output back to RGBA, then crops away the scaled letterbox area using the same contain rectangle. A 16:9 source can therefore produce the full 1280×720 result, while other aspect ratios produce the corresponding four-times crop. For a genuinely small source, that is a real fourfold enlargement of the model-space image. The phrase should not be misread as unlimited multiplication of any input. A 1920×1080 upload does not emerge at 7680×4320 from this implementation. The useful statement is precise: NanoVSR performs 4× reconstruction from its fixed 320×180 working representation.
05
Large sources are protected from an accidental downgrade
A fixed neural output creates an awkward case. If the source already contains more pixels than the cropped 4× model result, replacing it with that result would call itself enhancement while reducing resolution. The Worker detects this by comparing source area with model-output area. When the source is larger, it renders the neural crop back at the original width and height instead of shrinking the asset. It then blends 88 percent source color, 12 percent model output and a small protected-detail term derived from the source's local difference from a five-sample blur. This path keeps the original dimensions and most source information. It is conservative enhancement, not a second claim of 4× output.
06
Five-frame groups make video restoration temporal
Timeline Studio samples the retained source range at 12 frames per second and processes the frames in groups of five. For each group it seeks the video element to sourceStart plus the frame timestamp, creates five ImageBitmaps and transfers them to the Worker. A final group of two to four frames repeats its last bitmap to fill the T5 tensor. A final group with exactly one frame instead selects the T1 image model in this implementation. Each call returns only the requested number of outputs; the recorded sample documents this one-frame-tail behavior. Groups are non-overlapping in the current implementation. That keeps the control flow and memory bounded, though it also means an editor should pay special attention to visual continuity at five-frame group boundaries.
07
Twelve frames per second is an explicit product trade-off
The public dialog states that video is restored at 12 fps, and the restoration hook passes that value into the processor. The library clamps requested rate between one and fifteen frames per second. Twelve lowers the number of WebGPU inferences, decoded frames, PNG files and encoder inputs compared with preserving a 24, 30 or 60 fps source. It can be appropriate for short demonstrations, archival material or intentionally restrained motion, but it does not preserve the temporal smoothness of every source. The output duration remains tied to the retained source range, while fewer unique frames describe that duration. This is why full-speed playback matters: a crisp paused frame cannot reveal whether motion cadence is acceptable.
08
Trimmed clips begin at their actual source range
The processor reads sourceStart and sourceDuration from the selected segment instead of assuming the original file begins at zero. It limits duration to the media still available after that offset, calculates total frames from retained source duration and frame rate, then seeks every sample relative to sourceStart. This is the same non-destructive time discipline required elsewhere in a timeline editor. If a user trimmed ten seconds from the beginning before restoration, NanoVSR should enhance the visible clip, not silently spend minutes processing discarded footage. The audio composer receives the same source start and duration so picture and sound are cut from the same portion of the asset.
09
Model delivery is local-first, not model-free
Inference happens in a Worker on the user's GPU, but the application still needs model weights. Timeline Studio requests a pinned revision of either the T1 or T5 file from its model repository, choosing the preferred Hugging Face or ModelScope mirror according to settings and language. Download progress is exposed, and the ArrayBuffer is saved in a dedicated Cache Storage entry. Separate session promises are kept for image and video modes because their graph shapes differ. Later runs can report that the model came from cache before initializing WebGPU. Local processing therefore means source frames stay on the device; it does not mean the browser performs restoration without downloading software and weights.
10
Cancellation reaches the Worker and the encoder
Long restoration jobs need a real stop path. The hook owns an AbortController. Aborting a NanoVSR request posts a cancel message to the Worker and rejects the pending promise with an AbortError. The Worker records canceled request IDs, closes transferred bitmaps and suppresses progress or result messages for that request. During video composition, the same signal is passed into frame production and FFmpeg.wasm. If cancellation reaches encoding, the application terminates the FFmpeg instance and clears its cached loader. The original visual segment remains in place until a completed preview is explicitly applied. This is safer than swapping a half-produced asset into the timeline and hoping a later retry repairs it.
11
PNG frames become an H.264 MP4 with optional audio
Once each restored video frame is available as PNG, the shared browser compositor writes the sequence into FFmpeg.wasm's virtual filesystem. It encodes H.264 with the veryfast preset, CRF 18, yuv420p and faststart. When the selected source can be resolved to a Blob, it is added as an audio input, trimmed with the same source offset and duration, mapped as optional audio and encoded to AAC at 192 kbps. The shortest flag prevents one stream from extending beyond the other. The output is an MP4 Blob, and temporary frames, audio and output files are removed. This is a real video deliverable, not merely a high-resolution frame shown over the old clip.
12
The restored clip remains an editable project asset
Finishing inference does not immediately replace the current clip. The dialog presents synchronized source and restored playback, a timeline scrubber and a draggable comparison line. Apply creates a new user asset named with a nanovsr-4x suffix, records the output dimensions and duration, and swaps the selected segment only after approval. Enhancement metadata stores both original and processed source fields, including Blobs, dimensions and source timing, plus backend, frame rate and total frame count. If the same enhanced segment is reopened, the hook can recover the processed preview while retaining a path back to the original. Restoration becomes a reversible project decision rather than a destructive file conversion.
13
What the 644K model can and cannot improve
A compact model and fixed input are practical for browser GPU execution, but they do not make every source production-master quality. NanoVSR can strengthen edges, reduce the appearance of block compression and create more coherent texture from low-resolution evidence. It may also oversharpen, invent repeated patterns, alter small letters, simplify faces or produce unstable detail around fast movement and occlusion. Five-frame input gives temporal context, not a guarantee of perfect temporal consistency. A 12 fps output may be too sparse for sports, fast camera moves or high-frame-rate UI recordings. WebGPU support, adapter limits, cache space and FFmpeg memory remain device constraints. The right workflow starts with a representative short clip and accepts the result only after normal-speed review.
14
The useful claim is narrower—and better
Calling the feature 4× is correct at the neural model boundary: 320×180 becomes 1280×720 before aspect-ratio cropping. Product communication becomes misleading only when it implies that every source dimension is multiplied by four or that generated detail is recovered truth. Timeline Studio's code supports a more defensible claim. It uses T1 for images and singleton video tails, and T5 for the other video groups, protects higher-resolution sources, preserves the selected source range and available audio, exposes progress and cancellation, and returns a reviewable editable asset. That narrower explanation gives editors enough information to decide when browser-local super resolution is genuinely useful—and when the original footage is already the better source.
SOURCE REVIEW / MODEL AND RUNTIME
The 4× claim has shapes, hashes and a runtime path
This is an implementation record rather than a visual-quality benchmark. It identifies the exact model revision, tensor shapes, browser execution provider and temporal window used by the published feature.
REVIEWED SNAPSHOT
Timeline Studio commit 68980d1 · model revision d551be1
WHAT THE SOURCE SUPPORTS
The worker pins separate one-frame and five-frame FP16 ONNX graphs. Both accept 180 × 320 frames and return 720 × 1280 tensors, with FP32 browser I/O. Video is processed in five-frame groups through ONNX Runtime WebGPU; the model files are cached under a revisioned key.
HOW TO REPEAT THE CHECK
Record the source dimensions and duration, run one image and one short video on a WebGPU-capable browser, then inspect the reported backend, output dimensions, frame count and exported playback. Compare details at the same display size instead of judging the larger pixel count alone.
CLAIM BOUNDARY
A four-times tensor scale does not guarantee four-times perceptual detail. The current path normalizes inference to 320 × 180, uses a source-detail protection blend for already-larger inputs and has no cross-model accuracy benchmark in this record.
REFERENCE
Frequently asked questions
Does every video become four times wider and taller?
No. The neural working canvas scales from 320×180 to 1280×720 before aspect-ratio cropping. If the source is already larger than the effective model output, Timeline Studio preserves the source dimensions and blends protected source detail instead of shrinking it.
Does NanoVSR run locally?
Inference runs in a browser Worker through WebGPU. The pinned ONNX model must first download from a configured mirror and is then stored in Cache Storage.
Why does video use five frames at once?
The T5 model uses temporal context from five consecutive frames, giving it evidence that a one-frame image model cannot use.
What frame rate does the restored video use?
The current Timeline Studio workflow samples and composes restored video at 12 fps. The library accepts a clamped range from 1 to 15 fps.
Is the source audio preserved?
When the source can be resolved to a Blob with audio, FFmpeg.wasm trims the same source range and adds the optional audio stream to the H.264 MP4 as AAC.
Can it restore exact lost detail?
No. Super resolution reconstructs plausible high-frequency detail from low-resolution evidence. Generated texture should not be treated as forensic recovery.
Does applying restoration overwrite the original clip?
It creates a new PNG or MP4 asset and records the original and processed sources in enhancement metadata, so the editable project retains the original media path.
AUTHOR
Martin Delophy
Independent full-stack and algorithm engineer in China with 10 years of frontend, AI and audio/video development experience, including 5 years focused on AI. His open-source work covers browser AI, ONNX, WebGPU, Transformers, Stable Diffusion and local-first creative tools.
