#benchmark #anthropic #evaluation #openai

LLM Bench

Benchmark LLM providers side-by-side inside OpenClaw — measure latency, cost, accuracy, and quality across configurable prompt suites.

iriyaverma@iriyaverma

Install

openclaw plugins install clawhub:openclaw-llm-bench

openclaw-llm-bench

Created by Riya Verma (@iriyaverma)

Benchmark LLM providers side-by-side inside OpenClaw. Measure latency, accuracy, coherence, and token usage across configurable prompt suites — then compare results in a single table.

Install

bash

openclaw plugins install @riya/openclaw-llm-bench

Or place this folder under your OpenClaw extensions/ directory for local development.

What It Does

This plugin registers 8 agent tools and a CLI subcommand that let you:

Run benchmarks against any configured provider/model
Compare providers side-by-side on the same prompt suite
Track history of benchmark runs over time
Drill into details per-prompt to see where models excel or fail

Results persist to disk (~/.openclaw/bench-results/) so you can run benchmarks at different times and still compare.

Agent Tools

Tool	Description
`bench_run`	Run a benchmark suite or your own custom prompts against a provider/model
`bench_custom`	Quick one-off benchmark with your own prompts
`bench_create_suite`	Save a reusable custom suite with your own prompts
`bench_delete_suite`	Delete a custom suite
`bench_compare`	Compare latest results across all tested providers
`bench_suites`	List available prompt suites (built-in + custom)
`bench_history`	Show historical benchmark runs with optional filters
`bench_detail`	Per-prompt breakdown for a specific provider/model

Examples

Ask the agent:

"Benchmark openai/gpt-4o on the coding suite with 5 runs"
"Compare all my benchmarked models on the general suite"
"Show me the detailed breakdown for anthropic/claude-sonnet-4-20250514 on reasoning"
"What benchmark suites are available?"
"Test gpt-4o-mini with these questions: What is 2+2? What's the capital of France?"
"Create a suite called 'my-interview-qs' with these prompts: ..."

Prompt Suites

Suite	Prompts	Focus
`general`	10	Broad mix: reasoning, coding, math, creative, factual
`coding`	8	Easy through hard coding tasks, debugging, code review
`reasoning`	5	Logic puzzles, trick questions, formal deduction

CLI

bash

openclaw bench suites                     # list prompt suites
openclaw bench results                    # show all results
openclaw bench results -s coding          # filter by suite
openclaw bench compare -s general         # side-by-side comparison table
openclaw bench clear                      # wipe stored results

Configuration

In your OpenClaw config:

json5

{
  plugins: {
    entries: {
      "llm-bench": {
        config: {
          resultsDir: "~/.openclaw/bench-results",  // where to save results
          defaultRuns: 3,                            // runs per prompt per provider
          defaultSuite: "general",                   // default suite
          timeout: 60000,                            // timeout per call (ms)
        },
      },
    },
  },
}

How Scoring Works

Accuracy — If a prompt has an expected answer, responses are checked for exact or partial match. If a regex pattern is provided, it's tested against the response. For open-ended prompts, a heuristic scorer evaluates length, structure, and reasoning indicators.

Coherence — Measures structural quality: sentence count, word-length distribution, formatting (code blocks, lists), and repetition penalties.

Latency — Tracked per-run with P50 and P95 percentiles reported alongside averages.

Project Structure

text

index.ts              Plugin entry point — registers tools and CLI
src/
  types.ts            Shared TypeScript types
  suites.ts           Built-in prompt suites + custom suite lookup
  custom-suites.ts    Custom suite save/load/delete and ad-hoc builder
  runner.ts           Benchmark execution engine
  scoring.ts          Accuracy and coherence scoring
  storage.ts          Result persistence and comparison tables
  cli.ts              CLI subcommand registration

License

MIT