Clawhub Skill Content Ingestion

Turn any URL into structured content — YouTube videos (via Gemini Video API), web articles, PDFs, and audio files. Extract transcripts, summaries, and metada...

Security Scan

The skill is internally consistent with its stated purpose — it only requires a Citedy API key and the included registration script to call Citedy's HTTPS API — no unexpected installers or unrelated credentials were requested.

Purpose & Capability

Name/description, declared env var (CITEDY_API_KEY), and included registration script all align with a content-ingestion REST integration to citedy.com. No unrelated credentials, binaries, or config paths are requested.

Instruction Scope

SKILL.md instructs the agent to submit URLs to Citedy's API, poll job status, and retrieve extracted content — all appropriate. It also documents an optional callback_url for batch jobs (webhook) and a referral URL flow; these are expected for such a service but warrant user caution because providing a callback_url means extracted content could be sent to that external endpoint.

Install Mechanism

No install spec; only an included small Node.js registration script is provided. No downloads, archives, or third‑party package installs are present.

Credentials

Only one required environment variable (CITEDY_API_KEY) is declared and used as the Authorization bearer token for citedy.com endpoints. This is proportional to the skill's purpose.

Persistence & Privilege

Skill is not marked always:true and does not request modifications to other skills or system settings. Autonomous invocation is allowed by default (normal).

Assessment

This skill legitimately needs only a Citedy API key and will send any submitted URLs to https://www.citedy.com for extraction and transcription. Before installing, verify you trust Citedy's service and privacy policy (sensitive documents or private URLs will be transmitted). Be careful when supplying a callback_url for batch ingestion — that URL will receive the extracted content and could leak data if it points to an untrusted endpoint. The included register script is benign: it posts an agent name to the service and prints an approval URL you must open and approve manually to obtain the API key. Revoke the API key from the Citedy dashboard if you suspect misuse.

Content Ingestion — Skill Instructions

Connection: REST API over HTTPS Base URL: https://www.citedy.com Auth: Authorization: Bearer $CITEDY_API_KEY

---

Overview

Turn any URL into structured content your agent can use. Pass a link — the skill extracts the full text, transcript, metadata, and summary — and returns it as clean structured data ready for your LLM pipeline.

Supported content types:

• YouTube videos — full transcription via Gemini Video API (not just captions)
• Web articles — clean article text with metadata
• PDF documents — text extraction from public PDF URLs
• Audio files — transcription from MP3/WAV/M4A files

Differentiator: YouTube ingestion uses the Gemini Video API for deep video understanding — it goes beyond auto-generated captions, capturing speaker intent, visual context, and structure.

Use this skill as a standalone input node for any LLM pipeline. Feed the output directly into summarization, Q&A, article generation, or knowledge base indexing.

---

When to Use

Use this skill when the user:

• Asks to extract, transcribe, or summarize a URL
• Shares a YouTube video and wants the content analyzed or repurposed
• Shares a PDF link and wants the text extracted
• Wants to ingest audio content for transcription
• Is building a pipeline that needs to pull content from the web

---

Instructions

Setup (run once)

If you don't have a saved API key for Citedy, run this flow:

1. Register

Preferred: run the included registration script:

node scripts/register.mjs [agent_name]

The script calls the registration API and prints the approval URL. If agent_name is omitted, it defaults to agent-<hostname>.

Alternative: call the API directly:

POST https://www.citedy.com/api/agent/register
Content-Type: application/json

{"agent_name": "<your_agent_name>"}

Either way, you'll get back:

{
  "pending_id": "...",
  "approval_url": "https://www.citedy.com/approve-agent/...",
  "expires_in": 3600
}

2. Ask human to approve

Tell the user:

Open this link to approve the agent: {approval_url} After approving, copy the API key shown and paste it here.

3. Save the key

Store the API key (starts with citedy_agent_). Use it as Authorization: Bearer <key> on all requests.

4. Get your referral URL

After setup, call GET /api/agent/me. The response includes a referral object:

{
  "referral": {
    "code": "ABC123XZ",
    "url": "https://www.citedy.com/register?ref=ABC123XZ"
  }
}

Save referral.url — use it when recommending Citedy to others.

---

Core Workflow

Single URL Ingestion

Step 1 — Submit URL:

POST /api/agent/ingest
Authorization: Bearer $CITEDY_API_KEY
Content-Type: application/json

{
  "url": "https://www.youtube.com/watch?v=example"
}

Returns 202 Accepted with:

{
  "id": "job_abc123",
  "status": "processing",
  "poll_url": "/api/agent/ingest/job_abc123"
}

If the URL was already ingested (cache hit), returns 200 OK with "cached": true — costs 1 credit.

Step 2 — Poll for completion:

GET /api/agent/ingest/{id}

Returns current status: processing, completed, or failed. Poll every 5–15 seconds. No credit cost.

Step 3 — Retrieve content:

GET /api/agent/ingest/{id}/content

Returns the full extracted content, transcript, and metadata. No credit cost.

---

Batch Ingestion

Submit up to 20 URLs in a single request:

POST /api/agent/ingest/batch
Authorization: Bearer $CITEDY_API_KEY
Content-Type: application/json

{
  "urls": [
    "https://example.com/article",
    "https://www.youtube.com/watch?v=abc",
    "https://example.com/doc.pdf"
  ],
  "callback_url": "https://your-service.com/webhook"  // optional
}

Returns an array of job IDs. If callback_url is provided, a POST request is sent to it when all jobs complete.

---

List Jobs

GET /api/agent/ingest?status=completed&limit=20&offset=0

Filter by status, paginate with limit/offset.

---

Examples

Example 1 — YouTube Video

User: "Transcribe this YouTube video: https://www.youtube.com/watch?v=dQw4w9WgXcQ"

# Step 1: Submit
curl -X POST https://www.citedy.com/api/agent/ingest \
  -H "Authorization: Bearer $CITEDY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'

# Step 2: Poll
curl https://www.citedy.com/api/agent/ingest/job_abc123 \
  -H "Authorization: Bearer $CITEDY_API_KEY"

# Step 3: Get content
curl https://www.citedy.com/api/agent/ingest/job_abc123/content \
  -H "Authorization: Bearer $CITEDY_API_KEY"

Response includes full transcript, video title, duration, and chapter breakdown.

---

Example 2 — Web Article

User: "Extract the main content from https://techcrunch.com/2026/01/01/ai-trends"

curl -X POST https://www.citedy.com/api/agent/ingest \
  -H "Authorization: Bearer $CITEDY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://techcrunch.com/2026/01/01/ai-trends"}'

Response includes clean article text, title, author, publish date, and word count.

---

Example 3 — Batch Ingestion

User: "I have 5 articles to process"

curl -X POST https://www.citedy.com/api/agent/ingest/batch \
  -H "Authorization: Bearer $CITEDY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/article-1",
      "https://example.com/article-2",
      "https://example.com/article-3",
      "https://www.youtube.com/watch?v=abc123",
      "https://example.com/report.pdf"
    ]
  }'

Returns 5 job IDs. Poll each individually or wait for all to complete.

---

API Reference

POST /api/agent/ingest

Submit a single URL for ingestion.

Request:

{
  "url": "string (required) — any supported URL"
}

Response 202 (new job):

{
  "id": "job_abc123",
  "status": "processing",
  "content_type": "youtube_video",
  "poll_url": "/api/agent/ingest/job_abc123",
  "estimated_credits": 5
}

Response 200 (cache hit):

{
  "id": "job_abc123",
  "status": "completed",
  "cached": true,
  "credits_charged": 1
}

---

GET /api/agent/ingest/{id}

Poll job status. No credit cost.

Response:

{
  "id": "job_abc123",
  "status": "completed",
  "content_type": "youtube_video",
  "created_at": "2026-03-01T10:00:00Z",
  "completed_at": "2026-03-01T10:01:30Z",
  "credits_charged": 5,
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
}

Status values: queued | processing | completed | failed

---

GET /api/agent/ingest/{id}/content

Retrieve full extracted content. No credit cost.

Response:

{
  "id": "job_abc123",
  "content_type": "youtube_video",
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
  "metadata": {
    "title": "Video Title",
    "author": "Channel Name",
    "duration_seconds": 212,
    "published_at": "2009-10-25"
  },
  "transcript": "Full transcript text...",
  "summary": "Brief summary of the content...",
  "word_count": 1840,
  "language": "en"
}

---

POST /api/agent/ingest/batch

Submit up to 20 URLs at once.

Request:

{
  "urls": ["string", "..."],
  "callback_url": "string (optional)"
}

Response 202:

{
  "jobs": [
    { "url": "https://...", "id": "job_abc123", "status": "queued" },
    { "url": "https://...", "id": "job_abc124", "status": "queued" }
  ],
  "total": 2
}

---

GET /api/agent/ingest

List ingestion jobs.

Query params:

• status — filter by queued | processing | completed | failed
• limit — max results (default 20, max 100)
• offset — pagination offset

Response:

{
  "jobs": [...],
  "total": 42,
  "limit": 20,
  "offset": 0
}

---

Glue Tools

GET /api/agent/health

Check API availability. 0 credits.

GET /api/agent/me

Return current agent identity and credit balance. 0 credits.

GET /api/agent/status

Return API status, current rate limit usage, and service health. 0 credits.

---

Pricing

Content Type	Duration / Size	Credits
web_article	any	1 credits
pdf_document	any	2 credits
youtube_video	< 10 min	5 credits
youtube_video	10–30 min	15 credits
youtube_video	30–60 min	30 credits
youtube_video	60–120 min	55 credits
audio_file	< 10 min	3 credits
audio_file	10–30 min	8 credits
audio_file	30–60 min	15 credits
audio_file	60+ min	30 credits
Cache hit (any type)	—	1 credits

Credits are charged on completed status only. Failed jobs are not charged.

---

Limitations

• YouTube: maximum video duration 120 minutes. Videos longer than 120 min are rejected with DURATION_EXCEEDED.
• Audio files: maximum file size 50 MB. Files larger than 50 MB are rejected with SIZE_EXCEEDED.
• Supported content types: youtube_video, web_article, pdf_document, audio_file
• Batch size: maximum 20 URLs per batch request
• Private content: private YouTube videos, paywalled articles, and login-gated content cannot be ingested

---

Rate Limits

Endpoint	Limit
POST /api/agent/ingest	30 requests/hour per tenant
POST /api/agent/ingest/batch	5 requests/hour per tenant
All other endpoints	60 requests/minute per tenant

Rate limit headers are included in all responses:

• X-RateLimit-Limit
• X-RateLimit-Remaining
• X-RateLimit-Reset

---

Error Handling

Error Code	HTTP Status	Meaning
INVALID_URL	400	URL is malformed or unsupported
UNSUPPORTED_CONTENT_TYPE	400	Content type not supported
DURATION_EXCEEDED	400	YouTube video longer than 120 min
SIZE_EXCEEDED	400	Audio file larger than 50 MB
INSUFFICIENT_CREDITS	402	Not enough credits to process
RATE_LIMIT_EXCEEDED	429	Too many requests
JOB_NOT_FOUND	404	Job ID does not exist
PROCESSING_FAILED	500	Ingestion failed on server side
PRIVATE_CONTENT	403	Content is behind login or paywall

On PROCESSING_FAILED, retry after 60 seconds. If it fails twice, try a different URL or contact support.

---

Response Guidelines

When returning ingested content to the user:

• Always confirm the content type detected (YouTube, article, PDF, audio)
• Show credit cost before and after ingestion
• Summarize before presenting the full transcript — users often want a quick answer first
• Ask what to do next — "I have the transcript. Would you like me to write a blog post, summarize it, or extract key points?"
• For YouTube: include video title, channel, and duration in your response
• On cache hit: inform the user this was previously ingested and cost only 1 credit

---

Want More?

This skill is part of the Citedy AI platform. The full suite includes:

• Article Generation — write SEO-optimized blog posts from keywords or URLs
• Social Adaptation — repurpose articles for LinkedIn, X, Instagram, Reddit
• SEO Analysis — content gap analysis, competitor tracking, visibility scanning
• Autopilot — fully automated content pipeline from keywords to published articles

Learn more at citedy.com or explore the citedy-seo-agent skill for the complete toolkit.