{"skill":{"slug":"api-benchmark","displayName":"API Benchmark","summary":"Benchmark token generation speed across multiple LLM API providers. Measures TTFT (Time To First Token), tokens-per-second throughput, and total generation t...","description":"---\nname: api-benchmark\ndescription: Benchmark token generation speed across multiple LLM API providers. Measures TTFT (Time To First Token), tokens-per-second throughput, and total generation time. Use when comparing performance of different API providers, models, or testing API connectivity. Requires OpenCLAW config with provider definitions.\ncompatibility: Requires Python 3 with requests library. Reads target configuration from ~/.openclaw/openclaw.json. Supports anthropic-messages, openai-completions, and openai-responses API formats.\nmetadata:\n  author: Polar\n  version: \"1.0.2\"\n  requires:\n    config:\n      - ~/.openclaw/openclaw.json\n    env:\n      - OPENCLAW_CONFIG\n---\n\n# API Token Speed Benchmark\n\nThis skill benchmarks token generation speed across multiple LLM API providers.\n\n## When to use this skill\n\nUse this skill when you need to:\n- Compare token generation speed across different API providers\n- Measure latency and throughput of LLM models\n- Verify API connectivity and authentication\n- Test new API endpoints or models\n\n## How to run benchmarks\n\n### List available targets\n\n```bash\npython3 main.py --targets\n```\n\n### Run benchmark on a specific target\n\n```bash\npython3 main.py run --label <target-label>\n```\n\n### Run benchmark on all targets\n\n```bash\npython3 main.py run --all\n```\n\n### Run preflight check (verify API connectivity)\n\n```bash\npython3 main.py check --label <target-label>\npython3 main.py check --all\n```\n\n### Options\n\n- `-l, --label`: Specific target label to benchmark\n- `-a, --all`: Run on all available targets\n- `-r, --repeat`: Number of runs per prompt level (default: 1)\n- `-c, --category`: Run specific prompt category (can repeat: -c short -c medium). Options: short, medium, long\n- `-q, --quiet`: Quiet mode - suppress progress output\n- `--timeout N`: Request timeout in seconds (default: 120)\n- `--table`: Output as formatted table (default: JSON)\n\n## Configuration\n\nThe tool reads configuration from `~/.openclaw/openclaw.json`. Targets are defined in the `models.providers` section with:\n\n- `baseUrl`: API base URL\n- `apiKey`: Authentication key (or `${ENV_VAR}` to read from environment variable)\n- `api`: API format (anthropic-messages, openai-completions, openai-responses)\n- `models`: List of model configurations\n\n**Security Note**: Instead of hardcoding API keys in the config file, use environment variable placeholders:\n- `\"apiKey\": \"${ANTHROPIC_API_KEY}\"` will read from the `ANTHROPIC_API_KEY` environment variable\n\nExample provider config:\n```json\n{\n  \"models\": {\n    \"providers\": {\n      \"my-provider\": {\n        \"baseUrl\": \"https://api.example.com\",\n        \"apiKey\": \"sk-xxx\",\n        \"api\": \"openai-completions\",\n        \"models\": [\n          { \"id\": \"model-name\", \"api\": \"openai-completions\" }\n        ]\n      }\n    }\n  }\n}\n```\n\n## Output Metrics\n\n- **TTFT** (Time To First Token): Latency before first token arrives (seconds)\n- **TPS** (Tokens Per Second): Generation throughput\n- **Total Time**: Full generation duration (seconds)\n- **Input/Output Tokens**: Token counts from API usage data (or estimated at 4 chars/token if not provided by API)\n\nNote: Token counts are reported by the API when available. If the API doesn't return token counts, they are estimated at 4 characters per token.\n\n## Example Usage\n\n```bash\n# Check if a specific target is reachable\npython3 main.py check --label my-provider\n\n# Benchmark a single target\npython3 main.py run --label my-provider --repeat 3\n\n# Compare all targets\npython3 main.py run --all --table\n```\n","tags":{"latest":"1.0.2"},"stats":{"comments":0,"downloads":332,"installsAllTime":13,"installsCurrent":0,"stars":2,"versions":3},"createdAt":1771533156381,"updatedAt":1778491585715},"latestVersion":{"version":"1.0.2","createdAt":1771534862856,"changelog":"- Updated metadata to include required configuration and environment variables under a new \"requires\" section.\n- Removed explicit version field from metadata and adjusted the version number to \"1.0.2\".\n- No code or functional changes; documentation and metadata update only.","license":null},"metadata":null,"owner":{"handle":"polarjunction","userId":"s179tgd9hsqkzx2cmzr833j59n8848rz","displayName":"Polar","image":"https://avatars.githubusercontent.com/u/19593942?v=4"},"moderation":null}