Agent Tests

Define, run, and track tests for agent behavior. Test cases, assertions, regression tracking, and performance benchmarking. Zero external dependencies.

Stop guessing if your agent still works. Start testing it.

The Problem

Agent behavior changes with every model update, config change, and skill addition. Without tests, you have no way to know if something broke until a user catches it.

Agent Tests gives you a lightweight testing framework for agent behavior.

Quick Start

Add a test case

bash

node skills/agent-tests/agent-tests.js --test --add greet "Say hello" "Hello" "contains:Hello"

List all tests

bash

node skills/agent-tests/agent-tests.js --test --list

Run a specific test

bash

node skills/agent-tests/agent-tests.js --test --run greet

Run all tests

bash

node skills/agent-tests/agent-tests.js --test --run

Run performance benchmark

bash

node skills/agent-tests/agent-tests.js --benchmark greet 10

Runs the test 10 times, reports average/min/max duration and pass rate.

Check regression report

bash

node skills/agent-tests/agent-tests.js --regression

Shows pass rates, recent failures, and performance trends for all tests.

Remove a test

bash

node skills/agent-tests/agent-tests.js --test --remove greet

Status overview

bash

node skills/agent-tests/agent-tests.js --status

Assertion Types

Assertion	Example	What it checks
Default	"expected text"	Expected text is in output
contains:	contains:Hello	Output contains "Hello"
not_contains:	not_contains:error	Output does NOT contain "error"
regex:	regex:\d{3}	Output matches regex pattern
length:	length:42	Output is exactly 42 chars

Features

Test Management

• Add/remove tests with name, prompt, and expected output
• Track run count and pass rate per test
• Persistent storage in memory/agent-tests/tests.json

Assertion Engine

• Simple assertion syntax (no test framework needed)
• Supports contains, not_contains, regex, and length checks
• Extensible for custom assertions

Regression Tracking

• Groups results by test name
• Shows pass rate, total runs, and average duration
• Lists recent failures with specific assertion failures
• Flags tests below 80% pass rate with ⚠️

Performance Benchmarking

• Runs tests multiple iterations
• Reports average, min, max duration
• Tracks pass rate across iterations
• Saves benchmark history (last 50 runs per test)

Result History

• Stores last 1000 results per test
• Timestamps for every run
• Actual vs expected output captured

Configuration

Data files stored in: memory/agent-tests/

• tests.json — Test definitions
• results.json — Test results history (last 1000)
• benchmarks.json — Benchmark history (last 50 per test)

Override data directory:

bash

--dir /path/to/data

Agent Protocol

When testing agent behavior:

1. Add tests — --test --add for critical behaviors
2. Run before changes — --test --run before model/config updates
3. Benchmark periodically — --benchmark to track performance drift
4. Check regressions — --regression during heartbeats
5. Remove stale tests — --test --remove for outdated test cases

Limitations

• Test execution simulates output (actual agent output integration needed)
• Assertion engine is simple (regex, contains, length)
• No parallel test execution
• Results limited to 1000 entries

Comparison

Approach	Test Framework	Regression	Benchmark
Manual checking	❌	❌	❌
Full test framework	✅	⚠️	❌
Agent Tests	✅	✅	✅

Agent Tests gives you behavior testing + regression tracking + benchmarking with zero dependencies.

Design Principles

1. Zero setup — Works immediately, no config needed
2. No dependencies — Pure Node.js, no npm packages
3. Simple assertions — Easy to write, hard to misinterpret
4. Persistent — Tests and results survive restarts
5. Transparent — Every run is logged and reportable