Openclaw Skill Self Improvement

Health, eval, and regression system for continuously improving OpenClaw skills. Use when creating, auditing, or maintaining AgentSkills to ensure quality, de...

Security Scan

The skill mostly does what it says (local health checks and routing evals) but contains several incoherences and risky defaults — notably hardcoded absolute paths to another user's home, filesystem scanning of agent transcripts, and inconsistent script paths — so review before running on real data.

Purpose & Capability

The declared purpose (skill health, routing evals, heartbeat) matches the included scripts and outputs: the code scans SKILL.md files, runs eval cases, and writes reports. That said, the scripts assume and access additional system locations (e.g., /Users/m1/.openclaw/workspace, /Users/m1/.openclaw/agents, /Users/m1/.npm-global/...) which are not documented in SKILL.md and appear specific to the developer environment — this is an inconsistency that could cause the tool to scan unintended files on the host.

Instruction Scope

Runtime instructions and example commands in SKILL.md are reasonable, but the scripts perform broader actions than the examples explicitly call out: they recursively scan global system skills, and they read agent session transcripts (.jsonl) to infer usage signals. Reading transcripts is coherent with usage-tracking goals but is sensitive (may contain PII or secrets). Also, several path assumptions are inconsistent (examples run 'node scripts/…' while some scripts try to read files under workspaceRoot/skills/openclaw-self-improvement), increasing the chance of accidental or surprising behavior.

Install Mechanism

No install spec or external downloads are present; the package is instruction-plus-scripts only. All code is included in the repo (no network fetch or archive extraction). This is lower risk than arbitrary installers.

Credentials

The skill requests no environment variables or credentials, which is appropriate. However, it accesses local agent session files and system skill directories by default (hardcoded paths under /Users/m1). Access to transcripts and global node_modules is sensitive and should be justified and explicitly configured by the user rather than assumed by the scripts.

Persistence & Privilege

The skill does not request 'always:true' and does not modify other skills' configs. It writes reports under the provided workspace (.learnings/) which is expected. The main persistence/privilege concern is that it will read local agent transcript directories (sensitive) and system skill directories if those default paths exist.

What to consider before installing

This package appears to implement a local health-check and routing-eval system and includes runnable Node.js scripts — but review and test before running on real workspaces. Key things to check before use: - Inspect scripts locally (they are included) and confirm the paths they use. The scripts default to paths like /Users/m1/.openclaw/workspace, /Users/m1/.openclaw/agents, and /Users/m1/.npm-global/... — change these to point to a safe test workspace or pass an explicit workspace path when running. - Understand data access: the tool will scan SKILL.md files and will read recent lines from .jsonl agent transcripts to infer 'last used' signals. If your transcripts contain PII or secrets, run the scripts on a sanitized or isolated copy instead. - Address path inconsistencies: README/SKILL.md examples run 'node scripts/...', but some scripts (daily heartbeat, routing eval) look for assets under workspaceRoot/skills/openclaw-self-improvement. If you intend to run the packaged scripts, place them where the scripts expect or update the paths in the scripts to match your layout. - Run first in a small test directory or container with no sensitive data to observe outputs (.learnings/) and ensure behavior is safe. - Because daily-health-heartbeat uses child_process.execSync to invoke other scripts using constructed paths, verify those constructed paths resolve to the intended scripts in your environment to avoid accidentally executing unrelated files. If you want a cleaner install: move the repo into a dedicated workspace, update agentsRoot and systemSkillsRoot to point to intended locations (or make them configurable), and re-run. If you are not comfortable auditing JS, avoid running it on production systems or on a host that contains real agent transcripts.

A health, eval, and regression system for continuously improving OpenClaw skills.

Goal

Turn skill quality from an occasional cleanup task into a repeatable health loop.

What it does

1. Skill Health Check

Scans all workspace and system skills to detect:

• Duplicate skills: skills with highly similar names/descriptions
• Dark skills: skills with no recent usage signals
• Stale skills: skills unused for extended periods

2. Routing Eval

Runs evaluation cases against a keyword-weighted router to verify:

• Correct skills trigger for given inputs
• Wrong skills don't accidentally trigger
• Pass rate tracking over time

3. Daily Heartbeat

Automated daily run that:

• Executes health check + eval
• Compares against previous run
• Generates human-readable summary
• Surfaces changes and recommended actions

Usage

Run health check

node scripts/skill-health-check.mjs /path/to/workspace

Run routing eval

node scripts/routing-eval-runner.mjs /path/to/workspace

Run daily heartbeat

node scripts/daily-health-heartbeat.mjs /path/to/workspace

Output files

• .learnings/skill-health-report.json
• .learnings/routing-eval-report.json
• .learnings/daily-skill-health-summary.txt
• .learnings/skill-health-history.json

Current status

• Total skills scanned: 122 (workspace + system)
• Duplicate pairs: 1 (false positive)
• Dark skills: 0
• Eval pass rate: 100% (8/8)

Files

• scripts/skill-health-check.mjs — duplicate and dark skill detection
• scripts/routing-eval-runner.mjs — routing evaluation runner
• scripts/daily-health-heartbeat.mjs — daily automation
• references/routing-evals.json — evaluation case definitions

License

MIT