Install
openclaw skills install @vargai/varg-video-promptingHow to prompt varg's video, image, voice and music models so shots come out right on the first or second take: the working loop (references, look bible, environment master, character sheets, approved stills, audio lock, animate, QA), which model for which shot (Seedance 2.5, Kling 4, Veo 3.1, Wan 3, Nano Banana 2.1, GPT Image 2.5, ElevenLabs v4, Lyria, lip sync), prompt anatomy for video, stills and edits, what @image1, @video1 and @audio1 references copy, dialogue and voice rules, a frame and audio QA checklist, and gotchas from paid runs. Use when someone asks to "prompt Seedance", write a Kling or Veo prompt, pick a "varg video model" or asks "which model", does "image to video" or "video to video", needs a "consistent character" or location, makes "storyboard stills" or key frames, wants dialogue or lip sync, or asks why a clip looks like AI slop, drifted or ignored the prompt.
openclaw skills install @vargai/varg-video-promptingThis skill is what a team that makes video on varg every day learned the expensive way: which model to use for
which shot, how to write the prompt, what each reference actually copies, and how to check the result before
anyone sees it. It extends the basics in varg-ai (references/prompting.md has the shot, lens and light
vocabulary); it does not repeat them.
Calling the tools (sign-in, call_tool, jobs, uploads): references/mcp-tools.md.
Copy-paste prompts: references/prompt-templates.md. Per-model quirks:
references/models.md. Real runs and what they taught: references/examples.md.
estimate_cost on one real request, run one clip of each kind,
show it, wait for a go. Quote credits only, never money.Every step ends at a gate where the person looks at something cheap before anything expensive is made.
| Step | Make | Gate | Spend |
|---|---|---|---|
| 1. Brief | brief.md: format, length, platform, the one line to remember, references from real films and designers in refs.md (where to find them and how to skip the obvious pick: below), budget ceiling in credits, the person's own words verbatim with dates | person confirms | 0 |
| 2. Look bible | one look line (medium, palette, lens, light quality, grain), one identity paragraph per character, one place paragraph per location, a Never list of 5 things this look would never do | person confirms the words | 0 |
| 3. Environment master | one wide establishing still per location, no hero action, layout written as numbered elements. If several cameras share a set and geometry matters, a 3D blockout render first (threejs-animatic, varg-previz) | person approves the plate | a few stills |
| 4. Character sheets | front, 3/4, profile, back + a large face crop on flat grey, no text; product or vehicle sheets with a person for scale (varg-character-design) | person approves | a few stills |
| 5. Approved stills | one still per beat from master + sheets, on a numbered board (varg-mockups); fix faults here with surgical edits | every still OK at full size | ~6-16 credits each |
| 6. Audio lock | every line written with its pauses, breaths and actions (section 7); 2-3 auditioned voices per character; the music track with its beats measured and marked (beats.json); an SFX list with timestamps | person picks each voice and approves the track; the takes are frozen | voice lines (~84 each, ~42 on Turbo) and music (~11 on Lyria, ~25 on Mureka) |
| 7. Animate | one pilot per shot kind, then the batch, each run as long as its locked audio says; image-to-video from the approved still, or reference / video-to-video runs per location | pilot approved, batch quoted | the main cost |
| 8. QA | measured checks, then a frame sheet + audio check per clip against its still, written as a defect list (section 9) | every clip passes or is re-rolled | 0 (local) or ~6 per analyze_video |
| 9. Edit | cut, retime, lip-sync pass, text and graphics in code, loudness | person says ship it | lip-sync ~100 per shot |
Picture is cut to sound, not the other way round. Lock voice, dialogue and music before any clip is paid for: shot lengths come from the frozen takes and the beat grid (section 8), so each run is generated to a known length and every cut lands on a measured beat or the end of a line. A dialogue shot is still generated with sound on for the mouth (section 7); the locked take then replaces that sound by retime or lip-sync. Let the agent audition and shortlist voices; the person picks by ear.
References. Left alone, an agent picks the safe, average version of every look. Before the look bible, collect
real references: film stills from Film-Grab, ShotDeck or Shot Cafe, curated Pinterest boards; name the feeling,
skip the obvious first pick, and write down what each frame lends (palette, framing, lens, light, texture, motion).
The look bible's Never list keeps later prompts from drifting back to the average. Sources, search patterns and
the refs.md format: references/look-references.md.
Group shots by location: all coverage of one place comes from the same master and sheets, ideally one multi-shot generation with hard cuts, cut back into the timeline afterwards. A retake of one broken shot is a short run of just that shot, not the whole scene.
Credits are the public catalog's typical prices (varg-ai → references/models.md, October 2026); always confirm
with estimate_cost. Live list: list_models or https://varg.ai/models/llms.txt.
| Task | Model | Why | Credits / time |
|---|---|---|---|
| Hero shot with people; several shots of one place in one take; video-to-video from a previz | seedance_2_5 | best faces and physical performance; ≤9 image, ≤3 video, ≤3 audio refs; 4-30 s; obeys timed hard cuts | ~210 per 5 s 720p, ~480 at 1080p; 480p drafts billed ~16/s in our runs; 3-5 min |
| Animate an approved still faithfully; start-frame to end-frame moves; long calm single moves | kling_v4 | stays closest to the still (keeps everything in it, unwanted logos too); two files = start and end frame; native ambient audio | ~82 per 5 s 720p, ~123 at 1080p; 5-8 min per batch |
| Cheap motion test before paying for the hero model | minimax_h3_max_turbo, kling_v4_flash, or the hero model at 480p | same prompt, lower price; upgrade the winner | ~42 per 5 s |
| Best value text-to-video, lots of B-roll | wan_3 | top text-to-video rank, 2-30 s, refs; audio only through its provider option | ~53 per 5 s 720p |
| Short shots that fit 4, 6 or 8 s, 16:9 or 9:16 only | veo_3_1 (_fast, _lite) | strong image-to-video, first/last frame | ~105 silent, ~210 with audio per 5 s |
| Fastest top-tier clip, quick video edit | omni_flash_1_1 | ~30 s generation, 3-10 s | ~53 |
| Many short clips for a fast-cut visualizer | seedance_2_mini | cheapest Seedance, 720p, 4-15 s, strict moderation | ~74 per 5 s |
| Sheets, masters, scene stills, most edits | nano_banana_2_1 | ≤10 refs, holds identity, generates and edits | ~6 (1K), ~8 (2K), ~16 (4K) |
| Stills that must follow exact instructions or a blockout's geometry; dense text designs | gpt_image_2_5 | #1 for text-to-image and edits; follows "mapping" instructions | several times a Nano Banana still, depends on quality tier: estimate |
| Extend a still to a new aspect ratio, keeping its pixels | bria_fibo_edit_1_5_expand | outpaint without redrawing the subject | estimate |
| Voice-over and character lines | eleven_v4 (eleven_v4_turbo for drafts) | audio tags such as [whispers], 90+ languages | ~84 / ~42 per line |
| Song with vocals | lyria_3_5 | full songs, timestamped structure in the prompt | ~11 |
| Instrumental bed | mureka_9_5 | instrumental only | ~25 |
| Lip-sync onto a generated clip | sync_v3 (sync_v2_pro) | redraws only the mouth against the real voice | ~105 per short shot |
| Talking head from one still + audio | veed_fabric_1_0; long talks minimax_h3_max_lipsync | simplest path; lively but restrained on shouts | ~160-470; ~10-20 per s |
| Watch a clip and report what is in it | analyze_video (MCP) | timeline, transcript, on-screen text | ~6, same URL again is free |
Rules of thumb: Seedance for performance and multi-shot scenes, Kling 4 to move a frame you already love, a cheap
model to test motion, 480p or 720p for drafts, 1080p only for faces that stay on screen. More in
references/models.md; for Kling-specific prompting use kling-4-prompting.
Long, structured prompts (up to a few hundred words) worked on Seedance 2.5 when every sentence was a fact about the
frame. Kling 4 did best with a short director's prompt over strong key frames (kling-4-prompting). Write the blocks
in this order (full template in references/prompt-templates.md):
@image1 is the woman's character sheet: same face, short copper hair, green rain jacket. For a previz or rough clip: @video1 is a black-and-white grey-box previz. Copy from @video1 ONLY the camera, the hard-cut timing and the blocking. Do not copy its look; the result must not look like @video1.Where @image1 and @image2 differ in pose, light or setting, @image1 wins; @image2 only defines the person.Only one person in the room; every visible hand and sleeve belongs to her.
She has exactly two arms. Her hands are empty: no phone, no card, no glowing object at any time.3 shots, hard cuts: then 1. 0.0-1.8 s. Medium close-up from the left: she .... Times are
absolute from the clip start. One action per shot, a time for every event inside it (at 0.9 s the door slams).No dialogue, no lip-sync. plus the ambience you want, or the spoken line in quotes with its
delivery (section 7).Full-bleed 9:16 frame, no borders, no letterbox.No text, letters, numbers, logos or UI anywhere in the frame.Image-to-video from one approved still (any model) is shorter, and describes only motion:
Animate this exact frame. Locked-off camera, no zoom. After a short beat <one action>; <secondary motion: wind in
the grass, haze in the light beams>. Keep <the face, the prop, the composition> exactly as in the image.
No cuts, no new objects, no text.
Timing that works: 4-11 s per run (short runs obey better and retake cheaply); a 30 s take is four beats with six facts each (what, doing, where, camera, style, rules); spoken lines at about 2.5 words per second; generate a clip to start one beat before its cut so a late take still covers it.
@Image1 is <character> sheet: match face, hair and outfit exactly.
@Image2 is the master shot of <place>: same camera direction, same <place>.
<Place paragraph, with the props that must not change spelled out by count and position.>
Create one new single image, <shot size + angle> of the same scene: <pose, expression, action>.
Exactly <N> people; <anatomy constraints>.
Full-bleed image filling the entire frame edge to edge, no borders, no letterbox.
<Look line>. No text, letters or logos.
The edit prompt says what stays, then the single change:
Keep the FIRST image pixel-identical in every respect: camera, <list>. Add only one thing: <the change>.Keep this exact image: same room, camera, light, pose, clothes. Replace ONLY the man with <identity paragraph>. If the composition shifts, narrow it: Edit the first image only by swapping the person's head and face. Do NOT change the camera, framing, crop, symmetry, pose or clothes.Keep EXACTLY its camera position, lens, framing and perspective, and the position, size and orientation of every object. Do not add, remove or move any object. Mapping: grey blocks = <...>; the tall grey mannequin = the woman from the SECOND image; .... Give blockout people a head, arms and feet, or they come back
as logs or rope bundles.Remove only the logos and brand marks; change nothing else.Only these two people, nobody else.varg-video-editing). Do not regenerate the sheet to fix motion.files order: the first image is @image1, the first video @video1, the first audio
@audio1. Give each a role line that also says what it must not touch.@image2 is a frame from this same film: copy its drawing style, colours and set design. This fixed a segment
the reviewer called AI slop.From @video1 | Result |
|---|---|
| hard-cut timing | kept exactly |
| camera move (arc, push) | kept |
| blocking and action | kept |
| shot size | looser: a close-up came out as a medium shot, so frame one size tighter in the previz |
| the grey-box look | copied when no image refs were given (a mannequin became a toy-like person) |
| everything, once | a run came back as the previz itself, re-encoded; fixed by sending the ref in greyscale and saying "redraw every frame in full colour; the result must not look like @video1" |
So every v2v run needs an image ref for every character and product in frame. A prop or pose in the reference clip gets copied too: remove it from the reference instead of forbidding it in words.
@audio1 to drive performance and timing, never as the final sound.She leans in and whispers: "psst... wanna hear a secret?" A short pause. Then she shouts: "<line>", with delivery words (whispers, matter-of-factly, calling over wind). For no speech: No dialogue, no lip-sync, and name the ambience.eleven_v4 text alike:
A short pause. in a video prompt, [pause] in an eleven_v4 text). This also keeps the line inside
about 2.5 words per second (section 3).She sniffles. He swallows.
She exhales through her nose. The post's author called a sniffle between lines the biggest improvement by
far. Keep it small and in place; it is part of the delivery, not a new action.At 1.6 s he lifts his finger as if to say "one sec". Time it like
any other event and keep it the only big movement in the shot (one action per shot, section 3).astats RMS at 10-20 ms), and retime the video
piecewise (trim + setpts per segment) to the real audio. Body and lip timing then match well.sync_v3 (or sync_v2_pro) on the shot exactly as it plays in the edit, with only that shot's own vocal
slice. The output keeps the input length to the frame.search_voices (by intent in words, or filters) and audition 2-3 takes yourself or with
the person; choose by ear. If eleven_v4 answers voice_not_found for a name or preset id, pass the raw ElevenLabs
voice_id. Audio tags shape delivery: [breathless whisper] ... [fierce shout] ..., [calm, slow, soft-spoken],
[pause]. Transcribe each take (whisper_large_v3) and check brand words are pronounced cleanly.varg-music-video).
Never trust the BPM you asked for. Same lyrics, same "112 BPM" prompt: ElevenLabs Music measured 112.3 BPM and
50 s; Lyria came out near 97 BPM and 55 s, and the beat tracker first misread it as 129 (a triplet error) until a
drum-stem check corrected it. Pick the song by ear, then measure it.duration field (asked 20 s, got about 90 s) and exact rhythm instructions. Put the
length and timestamped sections in the prompt text (50-second ... [0:00-0:04] hook ... [0:46-0:50] outro. Song ends.), add Vocals from 0:00, no intro if needed, then trim on the grid.mureka_9_5, or instrumental: true on
generate_music). Describe the sound instead of naming people: ElevenLabs Music rejected a prompt that named a
composer as copyrighted material.Agents cannot watch video, but they can read frames and measure sound. Measure, don't grade yourself: the agent that made a clip is the worst judge of it, and "looks great" is not a check. Run the checks that need no opinion first (counts, frame scans, durations, transcripts, spelling, loudness, timings), then look, and write what you find as a defect list (time or frame, what is wrong, fix: cut, edit or retake), never as a score. Never ship a clip you have not checked.
Stills
Clips
ffmpeg -i clip.mp4 -vf "fps=2,scale=320:-1,tile=4x2" -frames:v 1 sheet.jpg for a 4 s clip; raise or lower fps with length), plus a frame at every intended cut.
Compare with the approved still or the previz at the same times.signalstats YAVG for sudden spikes (flash frames) and run freezedetect for frozen
stretches.silencedetect shows
where speech sits; transcribe and diff against the script.ffprobe duration and resolution match the request.The finished edit (measured first, then looked at)
ebur128 and normalise to the platform's target
(ecommerce-video-edit has a tested two-pass loudnorm to -14 LUFS).media-judge measures all of these, plus
upload banding with CAMBI: run it on the finished edit, in a fresh agent.Artifacts up to about 6 frames can be cut out in the edit; anything longer, or anything the reviewer sees twice, is a retake. Keep a jobs file (shot, job id, file URL, billed credits) the moment each job is submitted.
References and consistency
Prompt wording
Video
audio: false was not honoured on one Seedance route: the clip came back with a loud speech-like murmur. Also set
the provider's own audio flag for the route that runs (listed on https://varg.ai/models/seedance_2_5.md), and
strip audio in the edit (-an) for any clip that gets music or VO later.get_job_cost.cancel_job the slow one. In our runs cancelled and failed jobs were not billed; confirm
with get_job_cost.ffmpeg -sseof -1 -i clip.mp4 -update 1 last.png (each frame of the
last second overwrites the file, the last one stays); seeking to duration - 0.04 can miss it because the
container duration runs one frame past the last picture.Stills
get_tool("image", "gpt_image_2_5")) and pick quality explicitly.provider_options change, check the output's real pixel size and aspect: on a legacy Nano Banana
route a resolution option made the requested aspect ratio be ignored.Audio and voices
Operations
create_upload and a curl PUT to its proxy_url, or
upload_file with a public url (details in references/mcp-tools.md).curl; the file host rejected Python's default user agent with 403.$V[x] inside a string is an array subscript; write ${V}.| File | When |
|---|---|
| references/look-references.md | the brief and look bible: where to find film and design references, how to search past the obvious pick, refs.md, the Never list |
| references/prompt-templates.md | writing any prompt: bible lines, sheets, masters, scene stills, edits, i2v, multi-shot, v2v, retakes, dialogue, voice, music, lip-sync, ffmpeg helpers |
| references/models.md | choosing a model, durations, aspect limits, references, audio flags, quirks |
| references/mcp-tools.md | connecting the varg MCP and calling it: call_tool, schemas, estimates, batches, jobs, uploads |
| references/examples.md | real prompt, result and lesson from our own projects |
Related: varg-ai (setup, model catalog, basics), varg-character-design (sheets, realism, moderation-safe
wording), varg-consistent-characters (multi-scene ads with actors), varg-previz and threejs-animatic (3D
blocking, v2v packs), varg-mockups (still boards), kling-4-prompting, varg-music-video, varg-video-editing,
how-to-make-ai-video (find a ready skill for a format).