Guide · Media
An automatic editor is not one clever algorithm. It is a detector that proposes cut points, a scorer that ranks segments, and a cutter that stitches the winners together — and ffmpeg can do all three from the command line.
The discipline that keeps this fast is doing all the analysis on cheap passes and encoding exactly once, at the end. Every intermediate re-encode costs time and a generation of quality.
Hardcoding 1920x1080 at 30fps is how a pipeline breaks on the first phone clip someone hands you. Read it, branch on it.
ffmpeg's select filter exposes a per-frame scene score between 0 and 1, measuring how different a frame is from its predecessor. Thresholding that score gives you cut candidates.
0.4 is a starting point, not a constant. The right threshold depends on how much of the frame moves on a normal cut versus a normal pan, and that varies enormously by source.
| Footage type | Reasonable threshold | Why |
|---|---|---|
| Multi-camera edited source | 0.5 – 0.6 | Hard cuts are frequent and unambiguous; a low threshold fires on every pan and zoom instead |
| Talking head, single camera | 0.35 – 0.45 | Few genuine scene changes; this range still catches cutaways and reframes |
| Live event, locked-off wide shot | 0.5+, or skip it | Almost nothing changes visually — audio energy has to carry the whole selection |
| Sports, fast handheld | 0.25 – 0.35 | Motion and camera shake trigger false positives at higher thresholds, so tolerate more noise here and filter it downstream |
| Screen recording, tutorial | 0.6 – 0.7 | Long static frames punctuated by real transitions; a high threshold avoids firing on cursor movement |
For unedited footage — an event, a talk, a match — the visual track is often uniform and the interesting moments are audible: a cheer, a laugh, a sudden loudness jump. Audio energy is a crude but effective highlight detector, and it is cheap to compute.
# per-window loudness, one line per measurement ffmpeg -i input.mp4 -af ebur128=framelog=verbose -f null - 2>&1 # volume statistics across the whole file ffmpeg -i input.mp4 -af volumedetect -f null - 2>&1 # detect silence, then treat the gaps between silences as segments ffmpeg -i input.mp4 -af silencedetect=noise=-30dB:d=0.6 -f null - 2>&1Two approaches that work in practice. Peak-relative: compute short-term loudness over one-second windows, then keep windows more than roughly 6 dB above the file's median — those are your spikes. Silence-inverted: use silencedetect to find quiet gaps, treat the audible stretches between them as natural segments, and rank those segments by mean loudness.
Then pad each pick: a highlight starts before the spike. Two to three seconds of lead-in and one to two of tail turns a jarring fragment into a watchable clip.
Once both detectors have run, every candidate window gets one number so they can be ranked against each other and against the target runtime. A simple weighted sum works well in practice, because it lets you decide up front whether cuts or loudness should dominate the ranking for this footage.
score = w_audio × normalised_audio_energy + w_scene × normalised_scene_density Worked, two 10-second candidates, w_audio = 0.6, w_scene = 0.4: Segment A: peak loudness 6 dB over median → normalised_audio = 0.80 1 scene change in the window → density 0.10/s → normalised_scene = 0.25 score = 0.6(0.80) + 0.4(0.25) = 0.48 + 0.10 = 0.58 Segment B: peak loudness 2 dB over median → normalised_audio = 0.30 3 scene changes in the window → density 0.30/s → normalised_scene = 0.75 score = 0.6(0.30) + 0.4(0.75) = 0.18 + 0.30 = 0.48 Segment A wins despite fewer cuts, because w_audio was set high on purpose: for event footage, loudness is signal and rapid scene changes are usually camera shake, not content.The weights are the actual editorial decision, made explicit instead of buried in code. Event and talk footage should weight audio heavily, as above. A multi-camera edited source, where cuts were placed deliberately by a human, should weight scene density higher instead — the cut points themselves are already the signal.
Stream copy is enormously faster than re-encoding, but it can only cut at keyframes. Put -ss before -i and ffmpeg seeks quickly to the nearest keyframe; put it after, and it decodes up to the exact frame, which is accurate and slow.
The pragmatic compromise: cut with stream copy when the keyframe interval is small enough that a fraction of a second of drift does not matter, and re-encode only the clips where the cut point must be exact. Or force a short keyframe interval when you control the source encode.
A GOP (group of pictures) is the span between one keyframe and the next. Frames inside it are stored as differences from nearby frames, so a decoder — and a stream-copy cut — can only start cleanly at a keyframe. A long GOP means fewer keyframes, a smaller file, and worse worst-case drift when you cut with -c copy.
The trade is file size: a 1-second GOP writes roughly 8 times as many full I-frames as the 250-frame default, which grows the source file noticeably. Force a tight GOP only on footage you know will be cut heavily and stream-copied, not as a universal default.
The concat demuxer joins files without re-encoding, but only when every input shares the same codec, resolution, frame rate, and audio parameters. Mixed sources need the concat filter, which re-encodes.
# clips.txt file 'clip01.mp4' file 'clip02.mp4' file 'clip03.mp4' ffmpeg -f concat -safe 0 -i clips.txt -c copy reel.mp4If concatenated output plays the first clip and then freezes or desyncs, the inputs did not actually match. Normalise every clip to identical codec, resolution, frame rate, sample rate, and channel layout first, or switch to the concat filter and accept one re-encode.
Vertical delivery from horizontal footage is a framing decision, not a resize. Three options, in ascending order of effort:
| Approach | How it looks | When to use it |
|---|---|---|
| Blurred-background pad | Original centred, blurred enlarged copy behind it | Fast, safe default for talking-head and wide shots |
| Centre crop | Fills the frame, cuts off both sides | Subject reliably centred; loses anything at the edges |
| Tracked crop | Crop window follows the subject | Best result, needs detection per frame and real work |
Finish with loudness normalisation so clips cut from different parts of a recording do not jump in volume. The two-pass loudnorm filter targeting around −14 LUFS is the usual choice for social platforms.
The final encode is the one step in the pipeline that cannot be avoided, and it is the slowest one. Hardware encoders trade some quality-per-bit for a large speed win, which is usually the right trade while iterating and sometimes the wrong one for final delivery.
| Option | Platform | Speed vs software x264 | Quality per bit | Notes |
|---|---|---|---|---|
| VideoToolbox | macOS, including Apple Silicon | Roughly 3–8x faster | Slightly behind x264 at equal bitrate | No setup on a Mac; the sensible default while iterating |
| NVENC | NVIDIA GPU | Roughly 5–10x faster | Close to x264's medium preset on recent generations | Needs an NVIDIA GPU and current drivers |
| Quick Sync (QSV) | Intel CPU with integrated graphics | Roughly 3–6x faster | Historically behind x264, the gap has closed | Free hardware most Intel laptops already have |
| VA-API | Linux, various GPUs | Varies | Varies by driver | Least consistent of the options, driver-dependent |
| Software libx264 | Any CPU | Baseline | Best quality per bit — the reference everything else is measured against | Slowest; use it for the final export when quality matters most |
Three settings do almost all the work of an x264 or x265 encode, and confusing them is the most common way to waste render time or ship a worse-looking file than necessary.
| Setting | What it controls | Trade-off |
|---|---|---|
| CRF (constant rate factor) | Targets a visual quality directly; the encoder spends whatever bitrate that quality needs, scene by scene | The right default for most delivery. Lower CRF is higher quality and a bigger file — 18 is close to lossless, 23 is a reasonable default, 28 is visibly compressed |
| Target bitrate, typically 2-pass | Targets a file size or bandwidth ceiling directly | Use only when a platform enforces a hard cap. Quality varies scene to scene to hit the number instead of the number varying to hit quality |
| Preset (ultrafast … veryslow) | How much effort the encoder spends finding efficient encodings, at a fixed CRF | Slower presets produce a smaller file at the same visual quality, for more CPU time. veryfast is fine for iterating; use slow or veryslow only on the final export |
Every piece above is one stage. Chained, with the analysis kept cheap and the encode done exactly once at the end, the pipeline looks like this.
#!/usr/bin/env bash set -euo pipefail SRC="$1" # input.mp4 OUT="reel.mp4" WORKDIR=$(mktemp -d) # 1. probe DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$SRC") # 2. detect scene cuts (timestamps only) ffprobe -show_frames -of csv=p=0 \ -f lavfi "movie=$SRC,select=gt(scene\,0.4)" \ -show_entries frame=pkt_pts_time > "$WORKDIR/scenes.txt" # 3. detect audio spikes (silencedetect, inverted downstream) ffmpeg -i "$SRC" -af silencedetect=noise=-30dB:d=0.6 -f null - 2> "$WORKDIR/silence.log" # 4. score and select candidate segments against a target runtime python3 score_segments.py "$WORKDIR/scenes.txt" "$WORKDIR/silence.log" \ --target-duration 90 > "$WORKDIR/picks.txt" # 5. cut each pick, stream copy where possible i=0 while read -r start end; do i=$((i + 1)) ffmpeg -ss "$start" -i "$SRC" -t "$(echo "$end - $start" | bc)" \ -c copy "$WORKDIR/clip$i.mp4" echo "file '$WORKDIR/clip$i.mp4'" >> "$WORKDIR/clips.txt" done < "$WORKDIR/picks.txt" # 6. assemble: concat, then one encode at the end ffmpeg -f concat -safe 0 -i "$WORKDIR/clips.txt" -c copy "$WORKDIR/joined.mp4" ffmpeg -i "$WORKDIR/joined.mp4" \ -af loudnorm=I=-14:TP=-1.5:LRA=11 \ -c:v libx264 -crf 20 -preset slow -c:a aac "$OUT"The only genuinely custom piece is score_segments.py — the weighted-sum ranking from earlier, reading the two detector outputs and writing start and end timestamps for whichever candidates best fit the target runtime. Everything else is ffmpeg and ffprobe, called the same way a human would from a terminal.
| Symptom | Likely cause | Fix |
|---|---|---|
| Output plays the first clip then freezes or goes black | concat demuxer given inputs with mismatched codec, resolution, or audio parameters | Normalise every clip to identical parameters first, or switch to the concat filter and accept one re-encode |
| A -c copy cut lands noticeably off target | Stream copy snapped to the nearest keyframe | Re-encode just that clip, or shorten the source's GOP so the worst-case drift is smaller |
| Scene detection fires constantly on a locked-off shot | Threshold too low for low-motion footage; it is picking up compression noise | Raise the threshold per the calibration table above, or drop scene detection and rely on audio energy instead |
| Vertical crop cuts off the subject | Centre crop assumes the subject is centred in the original frame | Switch to the blurred-background pad, or implement a tracked crop |
| Loudness jumps between clips in the final reel | No normalisation pass, or loudnorm applied before concatenation instead of after | Apply two-pass loudnorm as the last step, on the assembled file, not on individual clips |
| Hardware-encoded output looks visibly worse than software at the same bitrate | Hardware encoders trade quality for speed at equal bitrate | Raise the target quality or bitrate for hardware encodes, or fall back to libx264 for the final delivery pass |
| ffmpeg reports an unsafe file name on concat | -safe 0 omitted, or clips.txt uses paths outside its own directory | Add -safe 0, or use absolute paths consistent with where clips.txt lives |
The honest counterpoint: an API gives you transcription, face tracking, and semantic highlight selection that a local ffmpeg pipeline does not have without extra models. If your selection logic genuinely needs to understand speech, run a local transcription model — the same local-over-API case, made generally, is in the local LLM guide — and keep the rest of the pipeline where it is.
This is the architecture behind EventReels — probe, detect, score, cut, assemble, one encode at the end, entirely on the machine, on the same stack the rest of this site runs on. It is the same no-third-party argument made more generally in the client-side tools guide, applied here to gigabytes of footage instead of a few kilobytes of JSON, and it slots into a broader shop workflow the way the automation guide describes.
FAQ
Use the select filter's scene score, which rates how different each frame is from the previous one on a 0 to 1 scale, and threshold it — for example select='gt(scene,0.4)'. Around 0.3 is sensitive and fires on camera movement, 0.4 is a reasonable default, and 0.5 or above catches hard cuts only.
Analyse audio energy rather than video. Compute short-term loudness with the ebur128 filter and keep windows several decibels above the file's median, or use silencedetect to find quiet gaps and rank the audible segments between them by mean loudness. Pad each pick with a few seconds of lead-in.
Use -c copy for stream copy, with -ss placed before -i so ffmpeg seeks quickly. The limitation is that stream copy can only cut at keyframes, so the cut point may drift by a fraction of a second; put -ss after -i and re-encode when the cut must be frame-accurate.
The concat demuxer with -c copy requires every input to share the same codec, resolution, frame rate, sample rate, and channel layout. When they differ, playback typically works for the first clip then breaks. Normalise all clips to identical parameters first, or use the concat filter and accept one re-encode.
Three options: pad with a blurred, enlarged copy of the video behind the original, which is the safe default; centre-crop to ih*9/16 by ih and scale to 1080x1920, which fills the frame but loses the edges; or track the subject and move the crop window, which looks best but requires per-frame detection.
No upload time on large files, no per-minute processing cost while you iterate on thresholds, no third party receiving footage of identifiable people, deterministic output you can debug file by file, and no exposure to another company's pricing or uptime. The trade is that APIs bundle transcription and face tracking that ffmpeg alone does not provide.
It depends on the footage, not a fixed constant. Multi-camera edited sources want 0.5 to 0.6 since cuts are frequent and unambiguous; a single-camera talking head wants 0.35 to 0.45; fast handheld or sports footage wants 0.25 to 0.35 to tolerate camera-shake noise; and a locked-off wide shot at an event should mostly rely on audio energy instead, since almost nothing changes visually.
CRF 23 is a reasonable default for libx264, with 18 close to lossless and 28 visibly compressed. Pair a low CRF like 20 with a slow preset for final delivery, where the encoder spends more time finding an efficient encoding, and use a fast CRF-23 pass with the veryfast preset while iterating on cuts, since encode time matters more than file size at that stage.