--- name: video-metadata-analyzer description: "Video content analysis pipeline — extract frames, transcribe audio, run LLM visual+audio analysis, synthesize structured Bilibili publish metadata (title, intro, tags, category, cover suggestion). Use when user says '生成投稿元数据', '视频元数据分析', or explicitly requests this skill. ⚠️ API modes send video frames and audio to external LLM providers — see Privacy section." privacy_notice: "API modes (vision-llm, audio-llm, cloud, api) transmit extracted video frames, audio, transcripts, and derived metadata to user-configured external endpoints. Videos may contain faces, voices, on-screen text, documents, or other sensitive data. Users should verify endpoint trust and data retention policies before processing. Use agent-direct or local modes for confidential media." version: 1.2.0 author: CyberKurry license: MIT homepage: https://github.com/CyberKurry/video-metadata-analyzer compatibility: "Requires ffmpeg, ffprobe, Python 3.8+. Optional: Pillow (frame compression), openai-whisper (local mode). External LLM API (OpenAI-compatible chat completions) for full analysis." platforms: [macos, linux] metadata: openclaw: requires: bins: ["ffmpeg", "python3"] primaryEnv: VIDEO_ANALYZER_LLM_KEY hermes: tags: [video, analysis, bilibili, transcription, vision-llm, publishing] category: media --- # Video Analyzer Three-stage video analysis pipeline: parallel visual + audio observation, then metadata synthesis for Bilibili publishing. ## When to Use - User says "生成投稿元数据", "视频元数据分析", "analyze video metadata" - User explicitly requests this skill for structured content analysis from a video file - User wants to prepare a video for Bilibili publishing (title, intro, tags, category, cover) - Upstream of `bilibili-publish-playwright`: this skill generates the metadata that feeds into Bilibili publishing > ⚠️ **Privacy Notice:** API modes (`audio-llm`, `cloud`, `vision-llm`, `api`) transmit video frames, audio, transcripts, and derived metadata to user-configured external LLM endpoints. Videos often contain faces, voices, on-screen text, or other sensitive data. Before using API modes, verify the endpoint trust, provider data retention policy, and your approval to process the media. Use `agent-direct` or `local` modes for confidential or regulated content. ## Architecture ``` Input Video ────────────────────────────────────────────────────── │ │ ├── Stage 1a: visual.py ──→ observations_visual.json │ │ (ffmpeg extract frames → encode → vision LLM observe) │ PARALLEL │ │ ├── Stage 1b: transcribe.py ──→ observations_audio.json │ │ (ffmpeg extract audio → transcribe + structure) │ │ │ └── Stage 2: analyze.py ──→ metadata.json ←───────────────────┘ (merge V+A observations → publishable metadata via LLM) ``` `run.sh` orchestrates: launches visual.py and transcribe.py as background processes (`&`), `wait` for both, then optionally runs analyze.py. ### Output Directory ``` $OUTPUT/ ├── observations_visual.json # JSON array: one object per frame ├── observations_audio.json # JSON object: transcript + structured info ├── metadata.json # (optional) Synthesized Bilibili metadata └── frames/ # (only with --keep-frames, auto-cleaned otherwise) ``` ## Procedure ### 1. Full pipeline (recommended — all external LLM API) ```bash bash scripts/run.sh \ --video VIDEO_PATH --output /tmp/va-out \ --transcribe audio-llm \ --audio-llm-key KEY --audio-llm-base URL --audio-llm-model MODEL \ --vision-llm-key KEY --vision-llm-base URL --vision-llm-model MODEL \ --max-frames 15 \ --synthesize-method api \ --analyze-llm-key KEY --analyze-llm-base URL --analyze-llm-model MODEL ``` ### 2. Agent-direct mode (no external API — agent reads frames/audio directly) ```bash bash scripts/run.sh \ --video VIDEO_PATH --output /tmp/va-out --keep-frames ``` Agent then reads `observations_visual.json` (placeholder frames), `observations_audio.json` (audio file path), and optionally the frame images + audio file directly to generate metadata. ### 3. Mixed / observe-only Omit `--synthesize-method` to observe only, then run `analyze.py` separately later. Each stage (visual, audio, synthesize) can use different keys and models. ### Key Parameters | Parameter | Default | Purpose | |-----------|---------|---------| | `--video PATH` | — | **Required.** Input video file | | `--output DIR` | — | **Required.** Output directory | | `--transcribe MODE` | `agent-direct` | `local` / `cloud` / `agent-direct` / `audio-llm` | | `--max-frames N` | `15` | Max frames per 4-min segment | | `--keep-frames` | `false` | Keep extracted frame images | | `--synthesize-method METHOD` | — | `api` / `agent` / `manual`. Omit = observe only | All `*-key`, `*-base`, `*-model` parameters follow the pattern: `--vision-llm-key`, `--audio-llm-key`, `--analyze-llm-key` etc. See [references/REFERENCE.md](references/REFERENCE.md) for the complete parameter table. ## Scripts | File | Role | |------|------| | `scripts/common.py` | Shared utilities: HTTP retry with backoff, media duration via ffprobe, JSON parse from LLM output | | `scripts/visual.py` | Frame extraction (auto-segment, auto-compress >200KB) + vision LLM observation. Long videos: segments processed in parallel (max 4 concurrent) | | `scripts/transcribe.py` | Audio extraction + transcription (4 modes). Auto-chunks large audio with 2s overlap for dedup | | `scripts/analyze.py` | Observations → publish metadata (3 methods: api/agent/manual). Heuristic fallback on API failure | | `scripts/run.sh` | Orchestrator: parallel visual+audio, then optional synthesis | ## Output Summary **`observations_visual.json`** — JSON array, one object per frame with `frame`, `objects`, `desc`, `texts`, `actions`, `style`, `cover_candidate`, `segment`, `segment_start`. **`observations_audio.json`** — `transcript`, `speakers`, `key_points`, `tone`. Agent-direct mode includes `audio_file` path. **`metadata.json`** — `title` (≤80 chars), `intro` (≤2000 chars), `tags` (≤10), `category` (B站 type2 平铺分区，30 个一级分区), `cover_suggestion` (primary + reason + secondary), `declaration` (6 选 1), `copyright_claim`, `watermark`, `author_marks`. ## Pitfalls - **API keys in chat**: Some platforms truncate keys with `…`. Always pass keys via command-line arguments, not through messages. - **Model capability**: Vision requires `image_url` support. Audio-LLM requires `input_audio` support. Check your provider. - **Game recordings**: Frames are large (~300-380KB vs ~30-80KB for phone). Auto-compression handles this, but plan rate limits for long videos. - **Long videos = parallel API calls**: 30-min video = 8 segments × 15 frames = 8 vision API calls (capped at 4 concurrent). Consider rate limits. - **Missing credentials auto-degrade**: Omitting LLM keys → preprocess-only or agent-direct mode. Scripts never crash on missing keys. - **`--interval` deprecated**: Ignored. Interval auto-calculated per segment based on `--max-frames`. - **Timeout protection**: `run.sh` auto-wraps with `timeout $VA_TIMEOUT` (default 3600s = 1h). Override via `VA_TIMEOUT` env var. ## Error Handling Three-layer defense: 1. **HTTP retry** — 3 retries with exponential backoff on 5xx / connection errors 2. **JSON parse retry** — 3 attempts with error feedback sent back to LLM 3. **Graceful degradation** — placeholder observations on visual failure, raw text on audio failure, heuristic fallback on synthesis failure ## Verification 1. Check exit code: `run.sh` returns 0 on success 2. Verify `observations_visual.json` has entries for expected frame count 3. Verify `observations_audio.json` has `transcript` field (non-empty for speech videos) 4. If `--synthesize-method` used, verify `metadata.json` has all required fields (`title`, `intro`, `tags`, `category`, `cover_suggestion`) For complete parameter reference, output schemas, standalone usage per script, and detailed error handling, see [references/REFERENCE.md](references/REFERENCE.md).