How to build a video editor from scratch

A timeline is a data structure, and the widget is downstream of it

The model everybody reaches for gives each clip a start, an end and a track, placed freely on a shared axis. That is the right model for compositing, where a title sits over footage and a music bed under both, so two things have to be allowed to occupy the same second. Every general purpose library is built this way because a general purpose library has to be.

A strip of adjacent scenes is a different object with one total rule: no gaps and no overlaps, ever. Under the free model every legal state of that strip is a coincidence maintained by whatever code ran last, so the drag handler clamps against the neighbour, absorbs the difference and decides what ripples. Then pointer capture is lost mid drag, a touch is cancelled, a second finger arrives, somebody nudges with the keyboard.

Model the boundary instead of the block. The grip between scene three and scene four owns a single number: moving it makes three longer and four shorter by the same amount, so a gap is not prevented, it is unrepresentable. The idea is not novel and that is the argument for it, because it is the gutter in Split.js, the connected handle in noUiSlider and the linked endpoints in waveform-playlist, each reached by somebody who got tired of the same handler.

Every invariant a model cannot express becomes an if statement in an event handler, and event handlers are where invariants go to die.

Time is one number, and everything else is a reading of it

Count the places a first draft keeps the playhead. The video element has currentTime. The playhead has a pixel offset. The ruler has a label, the selection has a scene index, the waveform has a cursor. Five copies of one fact, and they agree until the frame where they do not, which is the frame somebody screenshots.

None of the five is authoritative and they tick at different rates. The timeupdate event is not a clock: the HTML standard asks only that it fire periodically during playback, and browsers settle near four times a second. Setting currentTime is a request rather than an assignment, because the seeking algorithm may approximate for speed. To know which frame is actually on screen you need requestVideoFrameCallback, which reports the presentation time of the frame it just showed.

So give time one owner in the model and derive every other reading of it at the moment of reading. Position, label, selection and which block is lit become four functions of one number, wrong together or right together, which is the only kind of wrong worth having. The video element is then a view like any other, and the forbidden direction is the preview writing back: it reports where it got to, it does not define where you are.

What an edit costs, and why the answer belongs in a table

In a conventional editor the cost of an edit is a preview render measured in seconds, so nobody models it. In a generated video the cost is different in kind. The words came out of a script, the script was read aloud, the read set the timings, the timings set the beats, the beats decided the composition, and the composition was rendered. Rewrite one sentence and that whole chain behind it is false, at a price the person dragging cannot see.

Two instincts, both traps. Run everything again, which is always correct and always expensive. Or special case it, one conditional at a time, until a phase added a year later is invalidated by nobody, nothing fails, and the only symptom is the bill.

The structure that survives is boring: one row per kind of edit, naming the artefacts that edit destroys, with everything unnamed inherited. A phase added later is then a visible gap in one screen of data rather than an absence in control flow nobody rereads, and a test can assert the table covers every edit the interface can produce. Compute the impact before the person commits and show it to them, because a change that costs one clip and a change that costs the whole run should not look alike.

One refinement earns its keep. A change to the script belongs to the whole video, but a picture belongs to one scene, and a single table gets the second wrong in the expensive direction: adding one sentence is not a reason to fetch every stock clip and redraw every diagram.

The preview and the render are two programs, and they will disagree

A browser is very good at playing something approximately right, immediately: it composites on the GPU, drops what it cannot decode in time, and rasterises type the way the operating system does. A renderer takes the opposite trade, one deterministic file, one frame at a time, however long that takes. A dropped frame in a preview is a stutter nobody remembers. A dropped frame in a render is a wrong file somebody publishes.

That leaves two honest architectures and one common dishonest one. Either the preview is the renderer running in real time, sharing the drawing code and diverging only in timing, or it is an approximation whose divergence is documented, bounded and tested. The third option is two drawing paths that were identical in week one and a support queue full of screenshots where the text wrapped differently.

WebCodecs is what makes the first realistic in a browser: the decoder and the encoder exposed directly, frame by frame, so preview and export can walk one timeline with one set of code. It also hands back the bookkeeping the media element used to hide, since a decoded frame holds a real buffer that has to be closed and a queue that outruns your drawing exhausts memory in seconds. Playback and export want opposite queue policies, which is the seam where the two programs quietly become different again.

A dropped frame in a preview is a stutter nobody remembers. A dropped frame in a render is a wrong file somebody publishes.

Undo, and why a batch with one commit point beats a stack

The textbook answer is a command stack: every edit knows how to do itself and how to undo itself, pushed on the way in and popped on the way out. It works, and it costs. Every operation needs an inverse that stays correct in the presence of every other operation, so the bugs are never in the operations, they are in the pairs, and the pairs grow with the square of the feature list.

For a session that is bounded and ends in something expensive, hold the edited strip in memory as plain data, let a person change anything, and commit once. Nothing needs an inverse because nothing has been applied: undo is a comparison against the state the server handed you, and revert is that comparison taken all the way back.

This is not dodging the work. The commit point already exists, because the expensive act is the apply rather than the edit, and if twenty tweaks cost one render then the batch is the unit the person is already reasoning about. It pays again on cost: a batch is one diff, and one diff is one impact calculation, the union of what everything in it destroys, computed once from the two strips.

How Pipeline Video's scene editor does it

Under a finished video sits a strip of blocks, one per scene, each as wide as it is long and each showing that scene's own frame lifted from the video at its midpoint. Picking one opens it in a panel below, where the words are rewritten. They were inside the blocks in the first version, which made the field somebody writes a sentence in exactly as wide as that sentence was long, so the shortest scene got the narrowest box.

The three ideas above are the three it is built on. A grip between two blocks owns one number, so the strip cannot develop a gap or an overlap. One custom property carries where the video is, and the playhead's position, which block is lit and what the clock reads are three readings of it rather than three copies. Nothing leaves the browser until Apply, so the batch is the unit: one diff, one price, one job.

The cost model is those two tables, in a module that renders nothing, reads no file and calls no model, which is what lets the impact endpoint answer while somebody is still dragging. Twelve artefacts, seven kinds of change. Swapping one scene's picture destroys 3 of the twelve and leaves the script, the voiceover, the timings and the mix standing. Rewriting a sentence destroys 7. Adding a scene in the middle destroys 8, being a rewrite and a new picture at once.

And the part a post like this has every incentive to omit: today an apply renders the whole video again, every phase. The table is computed, shown before anybody commits and recorded on the job, but it states what an edit logically leaves alone rather than what the machine reuses. Two things stand in the way, neither small: the working directory is deleted the moment a render succeeds, and the renderer emits one file rather than one per scene. Until both change, nothing shown to a customer may read as a saving.

What to do about it

1. Choose the timeline model from the invariant, not from the widget. If contiguity is the rule, make a gap unrepresentable rather than defending it in a drag handler.
2. Give time one owner in the model and derive every other reading of it. The video element reports where it got to; it does not say where you are.
3. Write down what each kind of edit destroys as a table before writing the code that acts on it, and show the person that answer before they pay for it.