Explore capability boundaries of a target Skill, analyze optimization potential, generate an optimized version using Training-Free GRPO, and compile results into a structured report
All subsequent modifications will ONLY affect the optimized copy.
Step 0: Code Capability Detection (NEW v1.5.0)
Analyze scripts to detect their actual capabilities:
bash
cd subskills/code-capability-detector
python code_capability_detector.py <target-skill-path> <work-dir>
This analyzes:
Command-line parameters supported by scripts
Input/output formats
Dependencies
Error handling and validation presence
Expected outputs:
code_capabilities.json - Machine-readable capability data
code_capabilities.md - Human-readable report
Purpose: Ensures generated test tasks only use features the scripts actually support.
Verification: Confirm capability files exist before proceeding.
Step 1: Generate Test Tasks (sample-agent)
Execute the task generator:
bash
cd subskills/sample-agent
python task_generator.py <target-skill-path> ../..
The script generates:
12 base training tasks (6 standard + 6 advanced)
8 base test tasks (4 standard + 4 advanced)
If boundary probing is enabled and generates boundary tasks:
Training: 6 standard + 4 advanced + 6 boundary = 16 total
Test: 4 standard + 3 advanced + 3 boundary = 10 total
Expected outputs:
tasks/train/task_001/ through tasks/train/task_012/ (or task_016 with boundary tasks)
tasks/test/task_001/ through tasks/test/task_008/ (or task_010 with boundary tasks)
Each task directory contains: task.md, speccheck.md, and workspace/
Verification: Confirm all task directories exist before proceeding.
Step 2: Optimize the Skill (optimize-agent)
IMPORTANT: This step works on {target-skill}-optimized, NOT the original.
Execute the GRPO optimizer:
bash
cd subskills/optimize-agent
python grpo_optimizer.py <work-dir>/{target-skill}-optimized ../..
The script runs GRPO optimization with:
4 candidate variants per iteration
3-10 iterations with early stopping
SKILL.md and optional code-level optimization
All changes applied to the optimized copy only
Expected outputs:
{target-skill-name}-optimized/ directory containing the optimized SKILL.md
optimization_log.md documenting the GRPO optimization process
Verification: Confirm the optimized skill directory and log file exist before proceeding.
Step 3: Execute Both Skill Versions (exec-agent + Claude)
Part A: Generate Task Manifest
Execute the executor to generate task manifest:
bash
cd subskills/exec-agent
python executor.py <target-skill-path> ../..
Expected outputs:
task_manifest.json containing all tasks to execute
Part B: Execute Tasks via Skill Tool
Claude reads the manifest and executes each task using the Skill tool:
python
import json
manifest = json.load(open('task_manifest.json'))
for task in manifest['tasks']:
# Execute original skill
Use skill at manifest['target_skill_path'] with task['task_content']
Save output to task['original_result_dir']/output/
# Execute optimized skill
Use skill at manifest['optimized_skill_path'] with task['task_content']
Save output to task['optimized_result_dir']/output/
Expected outputs:
exec_results/original/task_001/ through exec_results/original/task_010/
exec_results/optimized/task_001/ through exec_results/optimized/task_010/
Each result directory contains: output/ with real skill execution results and run_log.md
Verification: Confirm all result directories exist with real outputs before proceeding.
Step 4: Failure Analysis (NEW v1.5.0)
Analyze failed tasks to identify root causes and suggest fixes:
bash
cd subskills/failure-analyzer
python failure_analyzer.py <work-dir>/exec_results/original <work-dir>
python failure_analyzer.py <work-dir>/exec_results/optimized <work-dir>
This analyzes:
Error messages and categorizes them (missing_parameter, missing_dependency, etc.)
Root causes of failures
Specific fix suggestions with code examples
Affected files and estimated fix difficulty
Expected outputs:
failure_analysis_original.json - Machine-readable failure data