Video Understanding

Use whenever the user provides any video URL, YouTube link, or video file, even if they only ask to watch, look at, react to, understand, analyze, summarize, transcribe, inspect, or extract answers from the video.

Video Understanding (Gemini)

Analyze videos using Google Gemini's multimodal video understanding. Supports 1000+ video sources via yt-dlp.

Requirements

• Python 3.10+ with uv
• yt-dlp — brew install yt-dlp / pip install yt-dlp
• ffmpeg — brew install ffmpeg (for merging video+audio streams)
• GEMINI_API_KEY environment variable

Trigger Rule

Use this skill first whenever the user provides a video URL, YouTube link, or video file. The trigger is the video input itself, not whether the user explicitly says "Gemini," "analyze," or "transcribe."

Casual asks like "watch this," "look at this," "what is this?," "you laugh," or "what happens here?" still require this skill.

Gotchas

• If the skill cannot run because GEMINI_API_KEY or another dependency is missing, report that blocker before falling back to local extraction.
• Requires yt-dlp and ffmpeg installed (sanity-check with which yt-dlp / which ffmpeg).
• Gemini File API uploads are reusable for follow-up questions while ACTIVE, but expire after about 48 hours.
• Gemini explicit CachedContent can speed repeated questions, but has model-specific minimum token counts and storage billing.
• The analyze script downloads non-YouTube videos locally first unless a valid cached Gemini file handle is reused.
• YouTube age-restricted or private videos require authentication cookies.

Default Output

Returns structured JSON:

• transcript — Verbatim transcript with [MM:SS] timestamps
• description — Visual description (people, setting, UI, text on screen, flow)
• summary — 2-3 sentence summary
• duration_seconds — Estimated duration
• speakers — Identified speakers

Usage

Analyze a video (structured JSON output)

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4"

Local video files work too:

bash

uv run {baseDir}/scripts/analyze_video.py "/path/to/video.mp4"

Ask a question (adds "answer" field)

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" -q "What product is shown?"

Continue questions against the same video

Use a stable --session-key for the conversation or Slack thread. --reuse-file-cache keeps and reuses an ACTIVE Gemini File API handle instead of re-uploading; stale handles automatically fall back to a fresh upload/download.

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache -q "Summarize this"
uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache --continue-chat -q "What happens after the pricing screen?"

For long videos with repeated follow-ups, add --use-context-cache to try Gemini explicit CachedContent. If Gemini rejects caching, the script falls back to File API reuse.

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache --use-context-cache --cache-ttl-seconds 3600 -q "Find all UI errors"

Override prompt entirely

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" -p "Custom prompt" --raw

Download only (no analysis)

bash

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --download-only -o video.mp4

Options

Flag	Description	Default
-q / --question	Question to answer (added to default fields)	none
-p / --prompt	Override entire prompt (ignores -q)	structured JSON
-m / --model	Gemini model	gemini-2.5-flash
--fallback-model	Optional fallback model for transient Gemini errors	none
-o / --output	Save output to file	stdout
--keep	Keep downloaded video file	false
--download-only	Download only, skip analysis	false
--max-size	Max file size in MB	500
--raw	Raw text output instead of JSON	false
--reuse-file-cache	Keep/reuse Gemini File API upload handles for follow-ups	false
--use-context-cache	Try Gemini explicit CachedContent for repeated questions	false
--cache-ttl-seconds	TTL for explicit CachedContent	3600
--cache-dir	Local directory for cache records	~/.openclaw/cache/video-understanding
--session-key	Conversation/thread key for follow-up reuse	none
--continue-chat	Include prior cached outputs as follow-up context	false
--purge-cache	Delete cached Gemini handles for this asset/session	false

How It Works

1. Local video files → Uploaded directly to Gemini File API
2. YouTube URLs → Passed directly to Gemini (no download needed)
3. All other URLs → Downloaded via yt-dlp → uploaded to Gemini File API → poll until processed
4. With --reuse-file-cache, valid Gemini file handles are reused for follow-up questions; expired/deleted handles trigger a fresh upload/download
5. With --use-context-cache, the script tries Gemini CachedContent and falls back to File API reuse if caching is unavailable
6. Gemini analyzes video with structured prompt → returns JSON
7. Temp local files are cleaned up automatically; remote Gemini uploads are deleted unless reuse/cache mode is enabled

Supported Sources

Any URL supported by yt-dlp: Loom, YouTube, TikTok, Vimeo, Twitter/X, Instagram, Dailymotion, Twitch, and 1000+ more.

Tips

• Use -q for targeted questions on top of the full analysis
• Use --session-key plus --reuse-file-cache for Slack-thread follow-ups so Loom/local videos are not re-uploaded every turn
• Use --continue-chat when the follow-up relies on prior wording like "that screen" or "the second speaker"
• Use --purge-cache when done with a sensitive video to delete the remote Gemini handles
• YouTube is fastest (no download step)
• YouTube follow-ups resend the public URL and optional prior context; there is no documented durable YouTube asset handle
• Transient Gemini 503/high-demand errors are retried; set --fallback-model only when that model has quota on the account
• Large videos (10min+) work fine — Gemini File API supports up to 2GB (free) / 20GB (paid)
• The script auto-installs Python dependencies via uv