experiment-lifecycle-governance

Add governance to experiment workflows — PIN-protected destructive ops, standardized metrics registry with thresholds, compare-scores ranking with gating, and competition rules audit. Builds on clearml-agent-dispatch and fysom-fsm-integration.

Overview

Governance layer for experiment workflows: protect destructive operations, standardize metrics, rank experiments with gating, and audit against competition rules.

Three sub-systems:

1. PIN Protection — 4-digit PIN guard for cancel/stop/delete operations
2. Metrics Registry — Standardized metric definitions with thresholds
3. Compare-Scores — Multi-model ranking with gating

Installation

bash

pip install expflow-pde

Version floor: install expflow-pde>=0.7.0 (public/PyPI line). The development line is ahead (0.7.3): anything marked "development line" below is not in the PyPI 0.7.0 artifact.

bash

uv tool install expflow-pde                  # 1) isolated CLI, `expflow` on PATH (recommended)
uvx --from expflow-pde expflow --help        # 2) run it without installing anything
uv pip install expflow-pde                   # 3) inside an existing project / venv
pip install expflow-pde                      # 4) no uv — pip and pipx both work
pipx install expflow-pde                     #    (pipx needs a recent version; 1.0.0 cannot parse specs)
# extras ride along with any form, e.g.: uv tool install "expflow-pde[clearml]"

All five forms above were run and verified (uv tool install / uvx --from / uv pip / pip / pipx).

1. PIN Protection Pattern

Architecture

text

~/.expflow/pin.hash          # SHA-256 hash of 4-digit PIN (never plaintext)
~/.expflow/experiments.jsonl # Experiment registry (each line = JSON record)

Module Design

python

# pin.py — 4 components:
# 1. init_pin(pin: str) -> hash          # Validate + hash + write
# 2. verify_pin(pin: str) -> bool         # Hash comparison
# 3. pin_is_set() -> bool                 # Check if PIN configured
# 4. guard(action_description) -> bool    # Interactive prompt

# sha256 hash — never store raw PIN
def _hash_pin(pin: str) -> str:
    return hashlib.sha256(pin.encode()).hexdigest()

# Validate exactly 4 digits
def _validate_pin(pin: str) -> None:
    if not pin.isdigit() or len(pin) != 4:
        raise ValueError("PIN must be exactly 4 digits (0-9)")

CLI Commands

bash

expflow pin init 1234          # Set PIN (SHA-256 stored)
expflow pin check              # Interactive verify
expflow pin clear [--force]    # Remove PIN
expflow pin status             # Show if active

# Guarded commands (require PIN unless --force):
expflow run cancel <id>            # Interactive PIN prompt
expflow run cancel <id> --force    # Skip PIN

2. Standardized Metrics Registry

Structure

python

STANDARD_METRICS = {
    "seg_total": {
        "type": "scalar", "group": "Score",
        "higher_is_better": True,
        "description": "Total segment score (primary competition metric)",
    },
    "pde_mean": {
        "type": "scalar", "group": "PDE",
        "higher_is_better": False,
        "threshold": 18.09,  # Competition gate
    },
    "train_time_min": {
        "type": "scalar", "group": "Time",
        "higher_is_better": False,
        "threshold": 60,  # Competition limit
    },
    # ... 13 total metrics across Score/Loss/PDE/Time/Model/Training groups
}

report_standard()

python

def report_standard(task: Any | None = None, **kwargs: float) -> dict[str, float]:
    reported = {}
    for name, value in kwargs.items():
        info = STANDARD_METRICS.get(name)
        if info is None:
            raise ValueError(f"Unknown metric '{name}'...")
        reported[name] = float(value)
        if task is not None:
            task.report_scalar(title=info["group"], series=name, value=float(value), iteration=0)
    return reported

3. Compare-Scores: Multi-Model Ranking

CLI

bash

expflow clearml compare-scores \
    --project PDEBench --tags task1 \
    --sort-by pde_mean --ascending \
    --gate pde_mean:lt:18.09 --gate train_time_min:lt:60

Gate Format

Gates use metric:op:value triplets:

• pde_mean:lt:18.09 — PDE mean < 18.09
• train_time_min:le:60 — Training time ≤ 60 min
• seg_total:ge:50 — Score ≥ 50

Operators: lt, le, gt, ge.

4. Competition Rules Audit

CLI

bash

expflow audit validate exp-001 --competition-rules --task-id abc123

Python API

python

from expflow_pde.audit import validate_competition_rules

result = validate_competition_rules(
    task_metrics={"seg_total": 57.09, "pde_mean": 15.0, "train_time_min": 45.5},
    task_params={"Args/--sub_step": "5"},
)
print(f"All pass: {result['all_pass']}")

Validation Checks

Check	Condition	Details
seg_total	Primary competition score (no gating)	Reported, not gated
pde_mean	Must be < 18.09	Threshold from STANDARD_METRICS
train_time_min	Must be < 60	Threshold from STANDARD_METRICS
sub_step parameter	Must exist and be > 0	Searches case-insensitive

Testing Patterns

PIN Tests (36 tests)

• Hash consistency: same input → same hash
• Validation rejects: wrong length, non-numeric, empty
• Init → file exists with correct hash
• Guard mock: correct → True, quit → False

Metrics Tests

• Registry structure: each metric has type, group, higher_is_better
• report_standard: returns dict of reported metrics

Compare Tests

• _apply_gate: all 4 operators (lt/le/gt/ge) with passing and failing cases

Experiment Loop Commands (0.7.0)

Group	Sub-commands	Purpose
expflow dispatch	status list tree stats audit-log archive	local SQLite dispatch database (what was submitted, when, by whom)
expflow iterate	run	one-shot iteration: diagnose → suggest → submit
expflow repeat	diagnose resubmit	repair diagnosis and resubmission (Hermes /goal compatible)
expflow hypothesis	open record list show rejected close	hypothesis tracking, including negative results

Pair them: dispatch gives the provenance trail, iterate/repeat close the loop, and hypothesis keeps the failed directions from being retried blindly.

Only in the development line (not in the 0.7.0 PyPI artifact yet)

• expflow competition — init stop status merge mask bootstrap collect validate (competition logging sessions, parameterized scoring).
• Modules added after 0.7.0: noise-aware champion validation + agent arbiter (validate.py), cross-session dead-end registry (registry.py), stagnation detection and experiment reports (monitor.py).

Hermes Agent Environment

• MCP server — expflow mcp starts a FastMCP server (18+ tools) for agent integration:

  yaml

  # ~/.hermes/config.yaml
  mcp:
    servers:
      expflow:
        command: "expflow"
        args: ["mcp"]
  

• Config — expflow init writes the toolkit config; real credentials (ClearML API, Langfuse keys, dataset paths) belong in .env / a local config.yaml, never in tracked files.

• Environment overrides — EXPFLOW_HOME, EXPFLOW_PIN_HASH, EXPFLOW_COMPETITION_DEADLINE, EXPFLOW_SEMANTIC_URL.

• Cost profile (not zero-cost) — the orchestration layer itself makes no LLM call, but runs cost real money and hardware: ClearML workers/queue time, GPU hours for the submitted experiments, and storage for artifacts. Quote costs from recorded runs, never an estimate.

Pitfalls

1. YAML/Env vs File Storage for PIN

PIN hash must NOT go into config.yaml (risk of git commit). Use ~/.expflow/pin.hash. Precedence: pin.hash file > .env EXPFLOW_PIN_HASH > config.yaml pin.hash.

2. get_last_scalar_metrics() clearml API

Returns nested dict: {"Score": {"seg_total": {"last": 57.09, ...}}, ...}. Flatten to {"seg_total": 57.09} for compare_scores.

3. --force Flag for Script Calls

Always provide --force / -f on PIN-guarded commands for CI/automation.

4. Interactive getpass vs Non-Interactive

getpass.getpass() works in terminals but fails in piped commands, CI, or subagent calls. Always provide --pin or --force as alternative paths.