research-harness

Cognitive discipline for AI-native scientific experimentation. Trigger when setting up controlled experiments with LLM agents, designing reproducible evaluation pipelines, or structuring research workspaces for long-running agent collaboration. Provides guardrails, not recipes — teaches agents how to reason about experiments, not which commands to run.

Version: 1.0.0 Cognitive discipline for AI-native scientific experimentation — guardrails, not recipes.

When to Use

Trigger this skill when the user:

• Sets up a new AI-native research experiment repo
• Designs controlled experiments with LLM agents
• Needs reproducible evaluation, statistics, and error analysis
• Wants to structure a research workspace for long-running agent collaboration
• Wants agent-safe research governance that prevents overclaiming
• Says anything like: "research harness", "experiment framework", "AI科研", "对照实验", "评分体系", "可复现性", "科研workflow", "agent协作科研", "可控实验", "效应量"

Core Philosophy

This skill does not prescribe what experiment to run. It prescribes how to think while running it.

Research agents fail not because they lack capability, but because they:

• Scale before validating the minimum loop
• Overclaim what the data proves
• Treat surprising results as methodology failures before checking the execution chain
• Delete failed runs to make progress look cleaner
• Change baselines or rubrics silently

The antidote is cognitive discipline — a set of non-negotiable mental habits enforced by repo structure, not by prompt reminders. Detailed reasoning for each discipline is in references/scientific-thinking.md.

---

Five Cognitive Disciplines

#	Discipline	Core Question	Deep dive
1	Minimum Closed Loop Before Scale	Can the smallest version produce distinguishable signals?	references/experiment-design.md
2	Isolated Variables & Attributable Baselines	Does each group add exactly one variable?	references/experiment-design.md
3	Dual-Track Validation	Do two independent scoring systems agree?	references/scoring-statistics.md
4	Effect Size Over Significance	What is the magnitude, not just the p-value?	references/scoring-statistics.md
5	Pipeline Before Interpretation	Was the execution chain verified before the hypothesis was questioned?	references/scientific-thinking.md

Disciplines 1-2: experiment design. 3-4: scoring & statistics. 5: critical reasoning.

---

Five Governance Rules

#	Rule	Principle
1	Human Owns Direction; Agent Owns Execution	Agent cannot change research questions, promote evidence without review, or make academic decisions
2	Evidence Has Status; AI Output Is Not Fact	All AI-generated evidence starts as candidate; only back-to-source verification promotes to verified
3	Failed Runs Are Data, Not Trash	Register every run in the manifest; failures are process evidence against survivorship bias
4	Protected Surfaces Change Only By Proposal	Baselines, rubrics, raw results, and schema require version bump + documented proposal
5	Every Handoff Needs an Alignment Doc	Short doc replaces long chat history for agent onboarding

Details in references/agent-collaboration.md.

---

Phase Workflow

Phase 0 · Scaffold

Goal: Set up the three-layer repo and root entry files.

• thinking-space/ — research direction, claims, decisions (human)
• execution-layer/ — briefs, logs, results, drafts (agent)
• code-workshop/ — runnable artifacts, packages

Root files: AGENTS.md (workspace map), PLAN.md (phase panel), WORKFLOW.md (procedure), harness/README.md (governance).

Directory skeleton and rationale: references/repo-architecture.md.

Phase 1 · Harden

Goal: Make the repo self-checking before formal execution.

1. Module contracts — Each core module gets a CONTRACT.md (purpose, inputs, outputs, invariants, local validator). Template in references/repo-architecture.md.
2. Local validators — scripts/validate_<module>.py per module; scripts/validate_repo_state.py as aggregator. Gate rule: 0 FAIL before any formal run.
3. Experiment manifest — experiments/results/manifest.csv as run-level provenance ledger (run_id, wave, task_id, group, model, version metadata, status, retry_of, git_commit).
4. Protected surfaces — Baselines, rubrics, raw results, scoring config, schema. Require version bump + proposal to change.

Phase 2 · Design

Goal: Design attributable controlled experiments.

• Progressive building: minimum artifacts → schema validation → small task set → dry run → scoring → expand. Design details in references/experiment-design.md.
• Controlled groups: Baseline → incremental treatments. Adjacent groups differ by exactly one variable.
• Gold checklists: Every task has must_include, forbidden, and scoring_notes.
• Output contract: Agent output follows a strict schema (YAML/JSON). The scorer and analysis pipeline depend on this contract.

Phase 3 · Execute & Analyze

Goal: Run experiments, score, compute statistics, analyze errors.

Preflight gate: local validators must pass. Then:

1. Dry run — print prompt, no API call
2. Smoke run — 1 task × 2 groups, verify output parsing
3. Wave 1 — small set × all groups, minimum viable data
4. Scoring: Track A (rule-based) + Track B (semantic) cross-validation. Details in references/scoring-statistics.md.
5. Statistics: Cohen's d primary, 95% CI, paired t, Wilcoxon. --reproduce flag for one-click reproducibility.
6. Error analysis: hallucination, output depth, specificity, task appropriateness.

Phase 4 · Handoff & Writing

Goal: Package results for the next phase or agent.

• Alignment doc: ~1 page with state, entry files, new surfaces, preflight commands, protected surfaces. Never pass chat history.
• Upstream proposals: Any insight affecting direction goes to sync/upstream_proposals/ first. Template in references/agent-collaboration.md.
• Writing markers: [REF-MISSING], [CRITICAL-CHECK], [TODO]. Never use AI numbers without verification.

---

Non-Negotiables

1. No unverified citation becomes a research fact
2. No debug result becomes a formal result
3. No agent changes baseline, rubric, or metric definitions without a proposal
4. No raw result is overwritten
5. No failed experiment is deleted
6. No phase gate passes before validators report zero FAIL

---

References

• references/repo-architecture.md — three-layer repo, module contracts, manifest, validators
• references/experiment-design.md — progressive building, controlled groups, gold checklists
• references/scoring-statistics.md — dual-track validation, effect size, reproducibility
• references/scientific-thinking.md — cognitive disciplines for agent-led research
• references/agent-collaboration.md — governance, evidence status, alignment docs