Tavily Search Pro Native Node

Research-grade Tavily toolkit for OpenClaw - native Node.js, zero dependencies. Search + Extract + Cache + Stats + Rate-limit backoff. Use when the user needs deep web research, multi-URL content extraction, or wants to track Tavily credit usage. Includes built-in response caching, full-page raw content, the Tavily /extract endpoint, automatic 429 backoff, a JSONL usage log, and a stats subcommand. Requires TAVILY_API_KEY in the process environment. For a minimal search-only skill, see tavily-search-native-node.

Research-grade Tavily toolkit. One script, four subcommands: search, extract, stats, cache.

Security behavior

• Reads TAVILY_API_KEY from the process environment only.
• Does not read credential files or ~/.openclaw/.env.
• Makes network calls only to Tavily's HTTPS endpoints: https://api.tavily.com/search and https://api.tavily.com/extract.
• Writes cache and usage logs only under ~/.openclaw/cache/tavily-search-pro-native-node/.
• Cache filenames are SHA-256 request hashes, not plaintext queries.
• Usage logs may contain plaintext search queries/URLs; use --no-log for sensitive calls.
• The ClawHub static-analysis potential_exfiltration warning is expected for this Pro skill because it combines env credentials, local cache/log file access, and Tavily network calls. It does not transmit local files or secrets.

Native Node.js. Zero dependencies. Small enough to audit directly.

When to use

Use this skill when:

• User needs thorough research, not just a quick lookup
• Multiple queries in a session (cache cuts credits 30-50%)
• User wants to extract full content from specific URLs
• User wants to know their Tavily credit usage
• Flaky network conditions (automatic retry)

Use tavily-search-native-node (the basic version) instead when:

• You just need one search and nothing else
• You want the minimal audit surface (no disk writes at all)
• You don't want caching or logging

Do NOT use this when:

• Privacy-sensitive queries (everything transmits to api.tavily.com)
• Simple URL fetches - use web_fetch for one-off page reads

How to run

The script is in scripts/tavily-pro.mjs.

Quick reference

# Search
node "<skill-dir>/scripts/tavily-pro.mjs" search "OpenClaw skills ecosystem"

# Extract full content from URLs
node "<skill-dir>/scripts/tavily-pro.mjs" extract https://example.com/ https://www.iana.org/help/example-domains

# See your usage
node "<skill-dir>/scripts/tavily-pro.mjs" stats

# Inspect or clear cache
node "<skill-dir>/scripts/tavily-pro.mjs" cache info
node "<skill-dir>/scripts/tavily-pro.mjs" cache clear

# Help
node "<skill-dir>/scripts/tavily-pro.mjs" help

Search subcommand

Flag	Values	Default	Purpose
--topic	general | news	general	news biases to fresh articles
--depth	basic | advanced	basic	advanced = 2 credits
--max	1-20	5	Results to return
--days	integer	7 (news only)	Age window
--include	comma list	(none)	Domains to include
--exclude	comma list	(none)	Domains to exclude
--raw-content	flag	off	Include full page text per result
--json	flag	off	Raw JSON output
--no-cache	flag	off	Skip cache
--no-log	flag	off	Skip usage log
--no-retry	flag	off	No backoff on 429
--ttl	seconds	24h general, 1h news	Override cache TTL

Extract subcommand

Wraps Tavily's /extract endpoint - takes one or more URLs, returns clean article text.

# Single URL
node "./scripts/tavily-pro.mjs" extract https://example.com

# Multi-URL (more efficient than separate calls)
node "./scripts/tavily-pro.mjs" extract https://a.com https://b.com https://c.com

# Advanced depth for pages with JavaScript rendering
node "./scripts/tavily-pro.mjs" extract --depth advanced https://spa-site.com

Flags: --depth, --json, --no-cache, --no-log, --no-retry, --ttl (default 7 days).

Credits: Tavily bills extract in batches of 5 URLs - 1 credit per 5 URLs (basic), 2 per 5 (advanced). Our log tracks this correctly.

Stats subcommand

Shows usage summary from the JSONL log.

node "./scripts/tavily-pro.mjs" stats            # last 30 days
node "./scripts/tavily-pro.mjs" stats --days 7   # last week
node "./scripts/tavily-pro.mjs" stats --json     # machine-readable

Output includes: total calls, searches vs extracts, cache hit rate, errors, credits used, and estimated credits avoided by cache.

Cache subcommand

node "./scripts/tavily-pro.mjs" cache info       # show dir, entry count, size, age
node "./scripts/tavily-pro.mjs" cache clear      # wipe the cache

What this skill does

• Reads TAVILY_API_KEY from the process environment only
• Calls https://api.tavily.com/search and https://api.tavily.com/extract
• Writes cached responses to ~/.openclaw/cache/tavily-search-pro-native-node/cache/
• Appends one JSON line per call to ~/.openclaw/cache/tavily-search-pro-native-node/usage.log
• Prints formatted, ASCII-safe human-readable results to stdout; --json preserves raw provider JSON

What this skill does NOT do

• Does not touch any files outside ~/.openclaw/cache/tavily-search-pro-native-node/
• Does not make network calls outside api.tavily.com
• Does not modify system configuration
• Does not auto-update
• Does not report to any external service

Caching

Responses cached to ~/.openclaw/cache/tavily-search-pro-native-node/cache/ keyed by SHA-256 of the request body + kind (search/extract). Different flag combos = different cache entries. No false hits.

Default TTLs:

• Search, general topic: 24 hours
• Search, news topic: 1 hour
• Extract: 7 days (web content is stable; re-extract only when freshness matters)

Override per-call with --ttl SECONDS. Skip entirely with --no-cache.

Rate-limit backoff

On HTTP 429 (rate-limited), the script retries up to 3 times with exponential backoff (1s, 2s, 4s) or respects the Retry-After header if present. Disable with --no-retry.

Network errors also retry. All other HTTP errors surface immediately.

Usage log

Every call appends one JSON line to ~/.openclaw/cache/tavily-search-pro-native-node/usage.log:

{"ts":1713732000000,"kind":"search","query":"OpenClaw","depth":"basic","topic":"general","cached":false,"credits":1}

Queries are logged in plaintext. If that's sensitive for a particular call, use --no-log to skip it.

Log is read-only by the script except for appends. Manually rotate/delete if it grows too large (typical: ~200 bytes/entry, so 1 MB = ~5000 calls).

Credentials

Requires TAVILY_API_KEY in the process environment.

Get a key: https://app.tavily.com - free tier is 1,000 credits/month.

Agent usage pattern

When invoking this skill:

1. Prefer one well-crafted search over several narrow ones
2. Use basic depth unless user asks for a deep dive
3. Use --include to scope to trusted domains when appropriate
4. Leverage the cache - don't disable it unless freshness matters
5. For follow-up deep reads of specific search results, pipeline: search -> pick URLs -> extract
6. Quote sources so the user can verify

Troubleshooting

• "TAVILY_API_KEY not set" -> export the env var in the process environment
• HTTP 401 -> key is invalid or revoked; regenerate at app.tavily.com
• HTTP 429 -> rate limited; script auto-retries up to 3x, then surfaces Retry-After
• HTTP 432 -> monthly credit cap hit; check usage dashboard
• Network timeout -> script auto-retries; persistent failures surface as error
• Stale results -> --no-cache for fresh, or cache clear to wipe all
• Cache dir growing -> cache info to see size, cache clear to reset
• Usage log too large -> rotate manually, e.g. PowerShell: Rename-Item usage.log usage-old.log; macOS/Linux: mv usage.log usage-old.log

Related

• tavily-search-native-node - the minimal basic version. No cache, no disk writes. Use when you only need search and want the absolute minimum audit surface.

Publishability notes

This skill is intentionally dependency-free and keeps cache/log writes under ~/.openclaw/cache/tavily-search-pro-native-node/. Before publishing or updating, run node --check scripts/tavily-pro.mjs, node scripts/tavily-pro.mjs help, node scripts/tavily-pro.mjs stats --json, and a no-key smoke test with a temporary home directory to verify credential errors without spending credits.