media-judge

Judge a finished AI-made video, image or track by measuring it, then rank takes on the defects found. One command reads every frame and writes facts with timestamps and evidence: broken frames, banding after a simulated X or YouTube upload, numbers that jiggle, spelling, too much on screen, nothing leading the eye, motion with no cause, pinned graphics, sliding labels, a face, outfit or light that changed, sound after the cut, loudness, voice against the script, and distance to references against the template. A fresh agent settles only what eyes can settle. No score out of 10. Use when someone says "review this video", "check my render", "self-review", "QA", "judge", "critique", "score these takes", "pick the best take", "does it match the reference", "it looks like a template", "is it ready to post", "reverse engineer this reel", or after any agent finishes a film, short drama, reel or ad. Not for making or repairing media (varg-ai, varg-video-editing) or ad law (ai-ad-compliance-check).

Step 9 of 10 in How to make a video with an AI agent in ten steps. The full guide, with every test and source: https://varg.ai/how-to/judge-an-ai-video-without-bias (as Markdown: add .md).

Measure, don't grade. A judgment here is a time, a number and a picture. "Looks great", "8/10" and "feels premium" are not findings, and neither is any adjective with no measurement behind it.

Code finds the moment and a fresh pair of eyes reads only that moment, in that order, because the other way round fails: 12 vision models scored at most 54.1% on spotting temporal glitches; Claude Opus caught 17.5% of them, Gemini 3 Flash said "glitch" to about 87% of clean clips (TempGlitch, gameplay clips). Uniform sampling at 0.5 fps caught 11.81% of faults under 0.2 s (SynFlash), and GPT-4 preferred its own output 0.705 and 0.912 of the time on tasks where people saw equal quality (Panickssery et al.). So:

1. Facts: judge.py reads every frame and every sample. It does not sample.
2. Eyes: a fresh agent opens only the evidence and answers narrow questions.
3. Verdict: counted defects. Ranking follows from the counts, never from a feeling.

What this has been tested on. Planted defects in footage it never saw while being tuned: 18 of 22 caught, 6 of 8 clean controls (25 of 25 and 10 of 10 on the footage it was tuned on). In a blind test on 19 anonymous clips, an agent with this skill found 12 of 14 planted defects, with no false claims on clean clips, using 116k tokens. An agent given only the failure list found 11 of 14, using 197k tokens, and reported the banding without an upload test. So the list itself does a lot. This adds repeatable numbers, the upload test, and fewer tokens. Both missed a face swap they weren't told to look for, so use --one-character.

1. Who judges

If you made this media, you do not judge it in this context. Start a fresh agent with no knowledge of the prompts or of how it was made, and give it only the file, this skill, the brief (what it must say, names, product) and the references. If you cannot start one, write self-judged at the top of the verdict.

2. Run the measurements

bash

uv run scripts/judge.py video film.mp4 --out runs/film \
  --allow "varg,Seedance"          # brand and product names for the spell check

Takes about a minute for a 12 s clip and about 5 minutes for a 47 s 1080p film on a busy laptop. Add what you know, because each flag turns a report into a test:

Flag	Use it when	What it turns on
--one-character	the brief says one person, or the same person across shots	face changes become flags. Without it a swap is only a number in neighbour_pairs, and a blind run of this skill missed one that way
--groups '[[0,1,2],[5,6]]'	these shots share one scene	outfit and light jumps become flags
--expected lines.txt	you know the on-screen text	every line must appear
--script voice.txt	there is a voice-over	word error rate, with --with faster-whisper
--refs ref.mp4	it copies a reference	pair sheet and distance per shot
--template tpl.mp4	it was built from a template	margin: closer to references or to template
--labels labels.json	you have the render's anchors	exact label drift against its target
--platform x|youtube|reels|tiktok|shorts	you know where it goes	upload simulation and safe zones
--size 1080x1920	a size was requested	size becomes a blocker
--music-only	no voice in the piece	skips voice tests

Images and tracks: judge.py image poster.png --out runs/poster [--ref ref.png] and judge.py audio track.mp3 --out runs/track.

Exit code 0 means nothing was flagged, 1 means flags to look at, 2 means the tool failed.

3. Read the report

Open runs/film/report.md. Each check is in one of four states:

• flagged: over its threshold. Open the evidence before you believe it.
• clear: measured and under the threshold. It says nothing beyond that one measurement.
• info: a number with no pass or fail, usually because nothing could be tested (no moving shot to test pinning, no counter on screen).
• not_run: a tool or model was missing.

A clear report does not mean a good video. "Not settled by this run" at the bottom of the report lists what is still open. references/checks.md explains each check: how it works, its threshold and its source, what false flags look like. Read the entry before overturning or trusting a flag on a check you have not used before.

Expect to overturn flags. On a real 47 s film we overturned all 4 spelling candidates we opened (OCR misreading small interface text), a subtitle measured while fading in, and a keyboard read as a pinned graphic. The upload banding and the unreadable screens were real.

4. Use your eyes on what code flagged

Work through questions in runs/film/facts.json. Each one names the frames to open and one narrow question. Read references/eyes.md first. The rules that change results:

• Open contact_sheet_*.jpg and each flagged evidence file. Do not judge from memory of the prompt.
• Ask "describe" and "what is unusual", never "what is wrong" (57.2% against 29.5% on GlitchBench).
• Answer per shot, with yes or no, a time, and something you can quote from the frame to prove you looked.
• Do not estimate where people look. Models score about 0.34 to 0.41 against real gaze. Use the heat map.
• Ask for observations, not verdicts, and let code decide (references/observe-prompt.txt). Never ask "which is better?" once: our judge picked the take shown second 75% of the time.
• A hunch without evidence goes under "Hints", never in the defect list.

Code cannot settle these, so eyes decide, using the measured moments: whether a move has a cause, whether a pinned graphic should be pinned, whether the most salient thing is the right thing, whether an outfit or light change is intended, and whether it looks like the references or just near them.

5. Challenge it

One pass, then stop (three at most).

• For each finding: what would make this not a defect? Is it in the brief? Is it OCR misreading a word? Is the "late sound" an unrelated noise near a cut? Open the frames again at source size. Keep the finding, or overturn it and write why.
• For each clear that matters: open one example. A check that stops firing looks exactly like a clean video.
• Stranger test: can you name three things in these frames that could only come from this brief? If not, report how many you found.
• Re-measure after a fix. A fix is not done until the same command stops flagging it, and nothing new appears.

6. Write the verdict

markdown

# Verdict: film.mp4
self-judged: no · ran 27 checks, 21 clear · not settled: lip sync, voice script

State: stopped (2 blockers)

| Severity | Time | Where | Measured | Evidence | Fix | Cost |
|---|---|---|---|---|---|---|
| blocker | 00:05.00 | whole frame | 2 black frames between lit frames | evidence/black_120.jpg | trim 2 frames at the seam | edit |
| major | 00:06-00:10 | dark gradient | CAMBI 4.1 source, 9.8 after upload at 2 Mbps | evidence/banding_3.jpg | add grain before encoding, or lift blacks | edit |

## Eyes
Shot 3, 00:07.20: label "Battery" points at the cell in the first frame and sits 90 px left of it in the last.

## Hints (no measurement behind these)
...

## Not checked
Mispronounced names. Whether the real X encoder matches the simulation.

Every row has a time, a number, a file and a fix. Cost is cut, edit or retake. Fix the cheapest first and name what stays as it is. Severity comes from SEVERITY in scripts/mj_rank.py. Change it there if a piece needs different rules, and say so.

Then save what eyes settled in runs/film/overrides.json, so it counts in the ranking and the reason stays on record:

json

{"overturned": {"spelling": "all three are OCR misreads, checked each crop"},
 "eyes": [{"severity": "major", "t": 7.2, "what": "label Battery no longer touches the cell"}]}

The score is the count. State is stopped with any blocker, needs a look with a major flag, no measured defect otherwise. ran 27, 21 clear is the only number out of something. Do not blend these into one mark.

7. Rank takes

bash

uv run scripts/judge.py rank runs/take1 runs/take2 runs/take3 --out runs/ranking

The order is blockers, then major flags, then minor flags, then the reference margin. ranking.md says why each take beats the next, and it counts each run's overrides.json. For takes tied on facts, judge each take alone with observations and let code decide, then use pairs in both orders where a flip stays a tie. See references/llm-judge.md for the protocol and how to write criteria.

8. Compare with a reference or a template

--refs aligns shots by picture and writes evidence/reference_pairs_0.jpg, ours next to theirs, worst pairs first, with distances for structure, palette, composition, texture and tone, plus shot-length rhythm. Open the sheet. Numbers say where it differs. Eyes say whether that matters.

With both --refs and --template you get margin = distance to template - distance to references. Above 0 means closer to the references. These distances are hand-built on OpenCV features. No study validates them for taste, so treat a small margin as a tie and look at the sheet.

9. Reverse engineer a finished piece

bash

uv run scripts/judge.py reverse ad.mp4 --out runs/ad

This writes a shot list from measurement: start, length, camera move and speed, shake, palette, lightness, where the focus sits, on-screen text, sound events, tempo, loudness, cuts per 10 s. Then open the sheets and add what code cannot measure: the hook, the lens, the light, the intent. Mark those as estimates. Nothing recovers the prompt or the seed. Say so.

10. Keep the judge honest

• tests/make_fixtures.py plants one known defect per clip in real footage, and tests/run_bench.py reports catches, misses and false flags. Run it after changing a check. Add a fixture every time a person finds something the judge missed.
• scripts/thresholds.json holds every threshold with status: published, convention or calibrated_on_fixtures. Only a few are published. Calibrate the rest on your own references and keep the labelled pass and fail beside them.
• Keep some checks out of any automatic loop, and look at a sample by hand. A judge an agent can optimize will get gamed: ranking takes on lip-sync score alone brought back lip smacking, and a still video scores perfectly on temporal consistency.

Limits

The attention map is a fast contrast, colour and motion model, far weaker than DeepGaze or ViNet. Use it as a report and swap in a better model if a decision depends on it. The upload test is a simulation, since X and YouTube do not publish their delivered encoders, so one private upload beats it. BT.1359's sync numbers were measured on lip sync, not on cuts. OCR misreads stylized type and misses text it cannot see. Whisper can repair a mispronounced word, so it hears what it expects. Code proves something is broken or correct, not that it is good.

Related

• references/checks.md: every check, threshold, source and false-flag pattern
• references/eyes.md: questions and rules for the vision pass
• references/llm-judge.md: LLM as a judge, bias measured on our own clips, the protocol, how to write a criterion
• references/ranking.md: gates, ties, pairs, and how not to get gamed
• references/build-your-own.md: how to build this yourself, step by step, with code that ran
• references/tools.md: other creators' tools and what each does better than this one
• prompt.md: paste-in prompts, for judging one video and for building your own judge
• code-to-video-previz scripts/judge_pack.py: pack and score a blind pair round
• varg-video-prompting section 9: the checklist used while making clips, varg-video-editing for fixes, video-moodboard for look checks before the render