OATDA Compare Models

Compare multiple LLM models (GPT-5, Claude Sonnet 4.5, Gemini 3, DeepSeek v4-pro, Grok 4, etc.) side-by-side in one parallel request via OATDA's unified AI gateway. Triggers when the user wants to benchmark models, evaluate quality differences, pick the best LLM for a task, run "gpt vs claude vs gemini" comparisons, do multi-model evaluation, or find the cheapest/best/fastest answer. Returns each model's output plus token usage and cost per model.

Send one prompt to multiple LLM providers in parallel and get each model's response back. Benchmarking, model evaluation, and "which LLM is best for X" use cases.

API Key Resolution

All commands need the OATDA API key. Resolve it inline for each exec call:

bash

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}"

If the key is empty or null, tell the user to get one at https://oatda.com and configure it.

Security: Never print the full API key. Only verify existence or show first 8 chars.

Model Mapping

User says	Provider	Model ID
gpt-5	openai	gpt-5
gpt-4o	openai	gpt-4o
claude, sonnet	anthropic	claude-sonnet-4-5-20250929
opus	anthropic	claude-opus-4-5-20251101
gemini	google	gemini-3-pro-preview
gemini-2.5	google	gemini-2.5-pro
deepseek	deepseek	deepseek-v4-pro
mistral	mistral	mistral-large-latest
grok	xai	grok-4-fast
qwen	alibaba	qwen3-max
kimi, moonshot	moonshot	kimi-k2.7-code
glm	zai	glm-5

Default comparison set if user doesn't specify: openai/gpt-5, anthropic/claude-sonnet-4-5-20250929, google/gemini-3-pro-preview, deepseek/deepseek-v4-pro.

⚠️ Models update frequently. If a model ID fails, query oatda-list-models for the latest available models.

API Call

Wichtig: Feld heißt modelId (nicht model!), models ist Array-of-objects, max 8.

bash

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}" && \
curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "<USER_PROMPT>",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 1024,
    "stream": false
  }'

Replace <USER_PROMPT> with the user's prompt (JSON-escape special characters).

Optional parameters:

• temperature: 0 (deterministic) to 2 (creative). Default: 0.7
• maxTokens: Max tokens per model response
• stream: true for SSE stream (default), false for batch JSON
• Per-model override: each entry in models may include its own messages array and service_tier (standard | flex | priority)

Response

Non-stream JSON shape:

json

{
  "results": [
    {
      "provider": "openai",
      "modelId": "gpt-5",
      "success": true,
      "response": "GPT-5's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 120, "total_tokens": 150, "cost": 0.0035}
    },
    {
      "provider": "anthropic",
      "modelId": "claude-sonnet-4-5-20250929",
      "success": true,
      "response": "Claude's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 95, "total_tokens": 125, "cost": 0.0021}
    }
  ]
}

Stream response: SSE events tagged with provider/modelId, one or more per model.

Present results side-by-side so the user can compare quality / cost / token-efficiency:

text

Comparison: "<prompt>"
─────────────────────────────────────────────────

[GPT-5]  (cost: $0.0035, 150 tokens)
GPT-5's answer...

[Claude Sonnet 4.5]  (cost: $0.0021, 125 tokens)
Claude's answer...

Errors

HTTP	Meaning	Action
400	Invalid request (bad modelId, empty array)	Check field names (modelId not model)
401	Invalid API key	Get one at https://oatda.com/dashboard/api-keys
402	Insufficient credits	Top up at https://oatda.com/dashboard/credits (or run oatda-check-balance)
422	Too many models (max 8)	Reduce models array length
429	Rate limited	Wait 5s and retry once

Individual model failures inside results[].success: false are NOT API errors — that specific model failed, the API call succeeded. Show the failure inline with the others.

Example

User: "Compare how GPT-5, Claude Sonnet 4.5, and Gemini 3 Pro explain recursion in one sentence."

bash

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}" && \
curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "Explain recursion in one sentence.",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 256,
    "stream": false
  }'

Tips

• Field name is modelId, not model (stricter schema than /api/v1/llm).
• Max 8 models per request (server-configurable max_compare_models).
• Cost accumulates per model: 4-model comparison ≈ sum of 4 individual calls. Run oatda-check-balance first if budget is tight.
• For a single-model call, use oatda-text-completion instead — simpler, returns direct cost.
• For deeper model discovery (capabilities, pricing, supported_params), run oatda-list-models first.
• NEVER expose the full API key in output.