Turning long footage into clips, automatically

The shape of the pipeline

1. Probe — read duration, resolution, frame rate, and audio layout so nothing downstream guesses.
2. Detect — find candidate boundaries with scene-change detection and candidate highlights with audio energy.
3. Score and select — rank the candidate segments and pick the ones that fit the target runtime.
4. Cut — extract the selected ranges, ideally without re-encoding.
5. Assemble — concatenate, convert aspect ratio, normalise loudness, encode once at the end.

The discipline that keeps this fast is doing all the analysis on cheap passes and encoding exactly once, at the end. Every intermediate re-encode costs time and a generation of quality.

Probe first

ffprobe -v error -show_entries \ format=duration:stream=width,height,r_frame_rate,codec_type \ -of json input.mp4

Hardcoding 1920x1080 at 30fps is how a pipeline breaks on the first phone clip someone hands you. Read it, branch on it.

Scene-change detection

ffmpeg's select filter exposes a per-frame scene score between 0 and 1, measuring how different a frame is from its predecessor. Thresholding that score gives you cut candidates.

ffmpeg -i input.mp4 \ -filter:v "select='gt(scene,0.4)',showinfo" \ -f null - 2>&1 | grep showinfo # print timestamps only, machine-readable ffprobe -show_frames -of csv=p=0 \ -f lavfi "movie=input.mp4,select=gt(scene\,0.4)" \ -show_entries frame=pkt_pts_time

• 0.3 — sensitive. Fires on camera movement and lighting changes as well as real cuts.
• 0.4 — a reasonable default for mixed footage.
• 0.5 and above — hard cuts only. Misses dissolves and slow transitions entirely.
• Single continuous shots — a locked-off camera at an event produces almost no scene changes, so this detector contributes nothing and audio has to carry the selection.

Finding highlights with audio

For unedited footage — an event, a talk, a match — the visual track is often uniform and the interesting moments are audible: a cheer, a laugh, a sudden loudness jump. Audio energy is a crude but effective highlight detector, and it is cheap to compute.

# per-window loudness, one line per measurement ffmpeg -i input.mp4 -af ebur128=framelog=verbose -f null - 2>&1 # volume statistics across the whole file ffmpeg -i input.mp4 -af volumedetect -f null - 2>&1 # detect silence, then treat the gaps between silences as segments ffmpeg -i input.mp4 -af silencedetect=noise=-30dB:d=0.6 -f null - 2>&1

Two approaches that work in practice. Peak-relative: compute short-term loudness over one-second windows, then keep windows more than roughly 6 dB above the file's median — those are your spikes. Silence-inverted: use silencedetect to find quiet gaps, treat the audible stretches between them as natural segments, and rank those segments by mean loudness.

Then pad each pick: a highlight starts before the spike. Two to three seconds of lead-in and one to two of tail turns a jarring fragment into a watchable clip.

Cutting without re-encoding

Stream copy is enormously faster than re-encoding, but it can only cut at keyframes. Put -ss before -i and ffmpeg seeks quickly to the nearest keyframe; put it after, and it decodes up to the exact frame, which is accurate and slow.

# fast, keyframe-aligned, no quality loss ffmpeg -ss 00:01:12 -i input.mp4 -t 8 -c copy clip01.mp4 # frame-accurate, re-encodes ffmpeg -i input.mp4 -ss 00:01:12.400 -t 8 \ -c:v libx264 -crf 18 -preset veryfast -c:a aac clip01.mp4

The pragmatic compromise: cut with stream copy when the keyframe interval is small enough that a fraction of a second of drift does not matter, and re-encode only the clips where the cut point must be exact. Or force a short keyframe interval when you control the source encode.

Concatenating the picks

The concat demuxer joins files without re-encoding, but only when every input shares the same codec, resolution, frame rate, and audio parameters. Mixed sources need the concat filter, which re-encodes.

# clips.txt file 'clip01.mp4' file 'clip02.mp4' file 'clip03.mp4' ffmpeg -f concat -safe 0 -i clips.txt -c copy reel.mp4

If concatenated output plays the first clip and then freezes or desyncs, the inputs did not actually match. Normalise every clip to identical codec, resolution, frame rate, sample rate, and channel layout first, or switch to the concat filter and accept one re-encode.

Converting to 9:16 vertical

Vertical delivery from horizontal footage is a framing decision, not a resize. Three options, in ascending order of effort:

# blurred-background 1080x1920 ffmpeg -i reel.mp4 -filter_complex \ "[0:v]scale=1080:-2,boxblur=20:2,crop=1080:1920[bg];\ [0:v]scale=1080:-2[fg];\ [bg][fg]overlay=(W-w)/2:(H-h)/2" \ -c:a copy vertical.mp4 # centre crop to 9:16 ffmpeg -i reel.mp4 -vf \ "crop=ih*9/16:ih,scale=1080:1920" -c:a copy vertical.mp4

Finish with loudness normalisation so clips cut from different parts of a recording do not jump in volume. The two-pass loudnorm filter targeting around −14 LUFS is the usual choice for social platforms.

ffmpeg -i vertical.mp4 -af loudnorm=I=-14:TP=-1.5:LRA=11 \ -c:v copy final.mp4

Why local beats an API for this

• No upload. A 4 GB event recording takes longer to upload on a home connection than to process locally. The transfer is the bottleneck, not the compute.
• No per-minute cost. Video APIs price per minute processed. Iterating on a threshold means reprocessing the same footage twenty times — free locally, expensive per call.
• No rights question. Footage of identifiable people at an event does not go to a third party at all, which removes both a consent conversation and a terms-of-service question about training use.
• Deterministic and debuggable. The same command produces the same output, and every intermediate is a file you can open.
• No dependency on someone else's uptime or pricing page. The pipeline works the same in three years.

The honest counterpoint: an API gives you transcription, face tracking, and semantic highlight selection that a local ffmpeg pipeline does not have without extra models. If your selection logic genuinely needs to understand speech, run a local transcription model and keep the rest of the pipeline where it is.

This is the architecture behind EventReels — probe, detect, score, cut, assemble, one encode at the end, entirely on the machine.

Tools referenced in this guide

• EventReels — the working auto-editing pipeline this guide describes.
• My stack — the rest of the tooling around it.
• Local LLM guide — the same local-over-API argument applied to language models.
• AI automation guide — where a pipeline like this fits into an actual workflow.