AI Video Repurpose × Digital Human Dubbing

AI video repurposing × digital human dubbing web tool. Paste a Douyin/TikTok link → wait 3 minutes → get a digital human video that looks like YOU said it. 5-step pipeline: download → transcribe → rewrite → TTS → digital human.

Core Scenario

Paste a Douyin/TikTok short video link → wait ~3 minutes → get a video that looks like YOU said it

Architecture

text

Paste link → [Download] → [Transcribe] → [AI Rewrite] → [TTS] → [Digital Human] → Deliver
    yt-dlp       Whisper       LLM        edge-tts     SadTalker

Quick Start

🏃 3-step setup

bash

cd ai-recreator

# 1️⃣ Install dependencies
pip install -r backend/requirements.txt

# 2️⃣ Configure (at minimum set your API Key)
cp backend/.env.example backend/.env  # then edit .env with your key

# 3️⃣ Start
cd backend && uvicorn main:app --host 0.0.0.0 --port 8080

Open http://localhost:8080 in your browser → paste a Douyin link → click "Start" → wait 3 minutes → get your digital human video.

Prerequisites

Dependency	Purpose	Install
FFmpeg	Audio/video processing	apt install ffmpeg or brew install ffmpeg
yt-dlp	Short video download	pip install yt-dlp (included in requirements)
PyTorch	Whisper/SadTalker inference	See official docs (CPU works but is slower)

5-Step Pipeline

Step 1: Download (modules/downloader.py)

• Downloads video & extracts audio via yt-dlp from Douyin/TikTok/Kuaishou
• Auto-detects platform; 120s timeout, 3 retries
• Supports proxy (DOWNLOAD_PROXY env) for cloud server bypass

Step 2: Transcribe (modules/transcriber.py)

• Whisper speech-to-text, default base model
• Model cascade: tiny → base → small → medium → large
• Cached per task to avoid re-transcription

Step 3: AI Rewrite (modules/rewriter.py)

• LLM rewrites short-video copy into natural, personal speech
• Strips influencer buzzwords ("hey guys", "OMG", etc.)
• Supports custom style prompts
• Falls back to rule-based rewrite when no API Key is set

Step 4: TTS Synthesis (modules/tts_engine.py)

• Default edge-tts (free, fast, great Chinese)
• Supports MeloTTS / CosyVoice switching
• Auto-splits long text + ffmpeg concatenation

Step 5: Digital Human (modules/digital_human.py)

• Default SadTalker: single photo → talking-head video
• Falls back to audio+placeholder video when SadTalker unavailable
• Wav2Lip interface reserved

API Reference

Method	Path	Description
POST	/api/tasks	Submit a creation task (video URL)
POST	/api/tasks/upload-video	Upload a video file directly (skip download)
POST	/api/tasks/upload-audio	Upload an audio file directly (skip download & transcribe)
GET	/api/tasks/{task_id}	Query task progress
POST	/api/tasks/{task_id}/confirm	Confirm rewritten text → start TTS
POST	/api/tasks/{task_id}/upload-video	Upload reference video for lip-sync
GET	/api/output/{task_id}/{filename}	Download generated video
GET	/	Web UI

Submit a task (URL)

bash

curl -X POST http://localhost:8080/api/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "video_url": "https://www.douyin.com/video/xxxxx",
    "custom_prompt": "Make it more casual and humorous",
    "tts_voice": "zh-CN-XiaoxiaoNeural"
  }'

Upload a video file (skip download)

bash

curl -X POST http://localhost:8080/api/tasks/upload-video \
  -F "file=@/path/to/your/video.mp4" \
  -F "custom_prompt=Use a friendly tone"

Configuration (.env)

Variable	Default	Description
OPENAI_API_KEY	""	For AI rewriting; leave empty for rule-based fallback
OPENAI_BASE_URL	""	Compatible OpenAI third-party API
OPENAI_MODEL	gpt-4o-mini	Model for rewrites
WHISPER_MODEL	base	tiny/base/small/medium/large
TTS_ENGINE	edge	edge/melotts/cosyvoice
DIGITAL_HUMAN_MODE	sadtalker	sadtalker/wav2lip/api
DOWNLOAD_PROXY	""	HTTP proxy for download (e.g. http://127.0.0.1:7890)

File Structure

text

ai-recreator/
├── README.md                 # Project overview
├── SKILL.md                  # This skill definition
├── skill-card.md             # ClawHub listing card
├── .gitignore                # Git ignore rules
├── cookies.example.txt       # Cookie template (optional)
├── backend/
│   ├── main.py              # FastAPI entry point
│   ├── config.py            # Config management
│   ├── models.py            # Data models
│   ├── task_manager.py      # Task queue manager
│   ├── pipeline.py          # Pipeline orchestrator
│   ├── requirements.txt     # Python dependencies
│   ├── .env.example         # Config template (copy to .env)
│   ├── assets/              # Static assets
│   │   └── default_avatar.png
│   └── modules/
│       ├── file_handler.py  # File upload handling
│       ├── downloader.py    # Video downloader
│       ├── transcriber.py   # Speech-to-text
│       ├── rewriter.py      # AI text rewriting
│       ├── tts_engine.py    # Text-to-speech
│       └── digital_human.py # Digital human generator
├── frontend/
│   ├── index.html           # Single-page Web UI
│   └── static/              # Static assets
├── references/
│   ├── faq.md               # FAQ (20 questions)
│   ├── anti-patterns.md     # Anti-patterns guide
│   ├── quickstart.md        # Quick start guide
│   └── privacy.md           # Privacy & data handling
└── data/
    ├── downloads/           # Download cache
    ├── audio/               # TTS output
    ├── transcripts/         # Transcription results
    ├── rewrites/            # Rewrite results
    ├── output/              # Final videos
    └── temp/                # Temporary files

Capability Boundaries

✅ What it does well

1. Short video repurposing - Extract speech from Douyin/TikTok/Kuaishou links, AI-rewrite into personal expression
2. Digital human talking-head generation - Voice + photo → lip-synced talking-head video
3. Style transformation - Removes influencer lingo, keeps core message
4. Multi-engine TTS - Supports edge-tts / MeloTTS / CosyVoice
5. Real-time progress tracking - Live 5-step pipeline status
6. Graceful degradation - Falls back to rule rewriting (no API Key) or placeholder video (no GPU)

⚠️ Requires user-provided assets

1. Voice cloning - Requires voice sample recordings (future release)
2. HD digital human (>720p) - Needs paid API (HeyGen/D-ID)
3. Lip-sync with original video - Needs Wav2Lip reference video (currently SadTalker single-image mode)
4. Custom digital human avatar - User needs to provide a front-facing half-body photo
5. Non-short-video platform URLs - Platforms other than Douyin/TikTok/Kuaishou need extra testing

❌ Out of scope

1. Batch processing - Single link per submission (submit multiple times for batch)
2. Video editing/cutting - Only extracts audio; no video editing
3. Watermark removal - Operates on audio track only
4. Deepfake/face swap - No face-swapping tech; lip-sync only
5. Live stream processing - No live stream or m3u8 support
6. Original content generation - Not responsible for factual accuracy of rewrites

🔒 Security & Privacy

1. URL filtering - Only http/https allowed; rejects file:// protocol injection
2. Timeout control - Each step has guard timeouts (120s~400s)
3. Graceful degradation - Handles missing API Key / GPU without crashing
4. File cleanup - data/temp/ cleaned after task completion
5. Data isolation - Audio & transcripts stored per task_id
6. API Key security - Read from .env, never logged
7. Lightweight dependencies - All AI capabilities optional; default runs fully local

👤 Audience

User Type	How to Use
Individual creators	Deploy locally, paste links via Web UI
Developers	Integrate via REST API
Teams	Deploy on internal server, share across members
Non-technical users	Use a deployed instance (if available)

Reference Docs

• references/faq.md — 20 FAQs + edge cases
• references/anti-patterns.md — 7 common anti-patterns + fixes
• references/quickstart.md — 10-second quick start
• references/privacy.md — Privacy & data handling

Version History

• v1.0.1 (2026-07-29): Fixed .env startup crash; added DOWNLOAD_PROXY; added --js-runtimes node; added upload-audio/upload-video endpoints; privacy docs & config template
• v1.0.0 (2026-07-28): Initial release. 5-step pipeline, Web UI, REST API, multi-engine TTS, SadTalker digital human, graceful degradation