SYSTEM DESIGN · LOCAL-FIRST

Building a Local-First AI Video Editor in the Browser

Timeline Studio is a React-based, local-first AI video editor. Voice synthesis, automatic captions, subject analysis, talking portraits and video export run as close to the user's device as the browser allows. The difficult part is not loading one neural network. It is coordinating inference, media state, rendering, caching and failure recovery so the result behaves like an editor rather than a model demo.

BY Martin DelophyPUBLISHED Jul 27, 2026REVIEWED Sep 4, 2026SOURCE-VERIFIED FIELD NOTETESTING METHOD

Building a Local-First AI Video Editor in the Browser visual guide

Local-first is a systems constraint

Keeping source media in the browser reduces unnecessary uploads and waiting, but moves cost onto the client. Large model downloads, browser storage, memory pressure, execution-provider differences and codec support all become product concerns. Timeline Studio therefore treats local-first as a default processing boundary: media and core computation remain on-device where practical, models load from immutable revisions on demand, and remote dependencies remain visible rather than being hidden behind a local label.

Use specialized runtimes and load them late

No single model path handles every media task well. Piper and VITS ONNX cover Chinese and multilingual speech, Kokoro 82M handles English voice generation, Whisper small q8 handles captions, YOLOS tiny and MODNet handle visual analysis, and JoyVASA plus LivePortrait drive talking portraits. Each path initializes only when the relevant feature is used. Warm sessions are reused inside the current visit so the editor avoids both a huge startup download and repeated graph initialization.

Workers define the responsiveness boundary

Inference, audio decoding, full-video analysis and encoding can each block the browser's main thread. Timeline Studio isolates ASR, vision, avatar generation, enhancement and source separation in dedicated Workers. The UI receives structured progress while remaining interactive. This boundary also makes cancellation and recovery explicit: canceling a full-clip enhancement leaves the original clip untouched, while a suspect neural frame can be retried without discarding the whole render.

Make the timeline the destination for AI output

An editor cannot stop at a download button. Generated speech must be inserted at the playhead, captions must remain linked to the matching audio, overlapping clips need additional lanes, and every change must be visible to preview and export. Timeline Studio represents AI output as assets plus clips in one timeline model. Voice duration comes from decoded audio, linked captions follow the same timing by default, and visual state is resolved at the requested project time.

Video intelligence must be temporal

Detecting a subject on the first video frame and applying that geometry to the entire clip fails as soon as the subject moves. The editor pre-analyzes video at adaptive intervals and stores timestamped YOLOS boxes and MODNet mattes. Preview, smart crop, caption avoidance, background removal and export resolve the record for the current source time. Longer clips use a wider sampling interval to keep inference time and memory bounded.

Version caches and design resilient export

The service worker uses cache-first behavior for large model files and network-first behavior for the app shell. Model URLs are revision-pinned so a cached graph cannot silently drift from runtime assumptions. Export follows a similar hierarchy: use a native browser format where possible, load FFmpeg WASM only when MP4 conversion is needed, and preserve a successfully rendered WebM if the final transcode fails. Browser AI becomes dependable when failure states are designed as carefully as the successful path.

Budget memory as a shared system resource

Browser memory is not an unlimited heap assigned to one model. Decoded video frames, audio buffers, ONNX weights, WebGPU tensors, canvas surfaces and encoder queues can exist at the same time. A feature that succeeds in isolation can therefore fail inside a real project. Timeline Studio reduces this collision by releasing temporary frame canvases, limiting analysis resolution, reusing sessions, avoiding duplicate media copies and keeping long-running jobs cancellable. Feature design also follows a memory hierarchy: a fast preview may use a smaller representation, while final export requests higher quality only when the preceding stage has finished. The interface reports model preparation separately from inference so users can distinguish a download from a stalled edit. Memory pressure is treated as a recoverable product state, not merely a console error: the original clip remains available, partially created output is discarded safely, and a smaller or more compatible path can be offered without corrupting the project.

Design observability for editors, not only developers

A local pipeline still needs evidence when something goes wrong. Each expensive operation exposes named stages such as downloading, initializing, analyzing, rendering and encoding, with elapsed time or item counts where those numbers are meaningful. Worker messages use structured payloads instead of arbitrary log strings, allowing the interface to show progress and the error boundary to retain actionable context. The same principle improves quality review. Timestamped detections, linked captions and export fallbacks make intermediate decisions inspectable instead of hiding them inside a model session. For developers, pinned model revisions and explicit execution providers make a bug reproducible. For editors, preserving the source and showing what changed makes experimentation reversible. This shared observability layer is what turns a collection of local AI features into a coherent application: users can understand the current state, cancel safely, retry deliberately and compare the generated result with the media that produced it.

SOURCE REVIEW / SYSTEM BOUNDARY

Local-first is checked component by component

A system architecture article becomes useful when its boundaries can be followed into code. This record links the orchestration, worker, model-source and render-plan layers instead of treating “local-first” as a blanket privacy label.

REVIEWED SNAPSHOT: Timeline Studio commit 68980d1 · reviewed September 4, 2026
WHAT THE SOURCE SUPPORTS: The application delegates heavyweight features to dedicated workers, selects revisioned model sources, keeps editable project state separate from media execution and builds explicit render plans. Remote catalogs and optional generation providers remain separate paths rather than being described as local inference.
HOW TO REPEAT THE CHECK: Open DevTools on a clean profile, preserve the Network log, load one local workflow and record page, model and worker requests. Then activate a remote catalog or generation provider separately. Compare destinations and request bodies, inspect browser storage, and clear site data before repeating the run.
CLAIM BOUNDARY: Local-first is not offline-by-default and is not a promise that no request occurs. Application assets, fonts and model weights must download; optional providers can cross a remote boundary; browser extensions and analytics require separate inspection.

REFERENCE

Frequently asked questions

Does local-first mean the editor is fully offline?

Not initially. Application and model assets must be downloaded, but repeat sessions can reuse cached resources and source media does not need to be uploaded for the core workflow.

Why use several models instead of one multimodal model?

Specialized models offer better control over quality, memory, execution backends and lazy loading for each task.

Why must preview and export share the timeline model?

Separate state paths drift over time and create results that differ from what the user saw while editing.

Where can I try the editor?

Timeline Studio is available at video-editor.ai-creator.top and its source is linked from the project page.

AUTHOR

Martin Delophy

Independent full-stack and algorithm engineer in China with 10 years of frontend, AI and audio/video development experience, including 5 years focused on AI. His open-source work covers browser AI, ONNX, WebGPU, Transformers, Stable Diffusion and local-first creative tools.

About the author →