Use when the user wants to test an agent skill on a remote jump host after updating it locally. Triggers on "test skill remotely", "remote test", "远程测试 skill", "到跳板机上测试", "验证 skill 更新效果". Does NOT run the target skill locally — only orchestrates remote execution and report comparison.
Orchestrate end-to-end testing of agent skills on a remote jump host. After
the user updates a skill locally, this skill SSHs to the remote host, updates
the installed skills, runs the target skill via opencode run in a dedicated
test directory, retrieves the generated report, and compares it against the
previous run to verify that changes match the skill update.
Output Language Rule
Detect the language of the user's conversation and use the same language for all output.
Chinese input -> Chinese output
English input -> English output
Prerequisites
Required tools:
ssh — access to the remote jump host
scp — for retrieving reports from the remote host
npx — installed on the remote host (for npx skills add)
opencode — installed and configured on the remote host
Workflow
dot
digraph test_flow {
"Collect SSH config\n+ skill name + prompt" [shape=box];
"SSH: create test dir" [shape=box];
"SSH: install skills\n(project level in test dir)" [shape=box];
"SSH: opencode run\n(output to log)" [shape=box];
"SCP: retrieve report + log" [shape=box];
"Find previous report" [shape=diamond];
"Compare reports\nvs SKILL.md changes" [shape=box];
"Output analysis" [shape=box];
"Collect SSH config\n+ skill name + prompt" -> "SSH: create test dir";
"SSH: create test dir" -> "SSH: install skills\n(project level in test dir)";
"SSH: install skills\n(project level in test dir)" -> "SSH: opencode run\n(output to log)";
"SSH: opencode run\n(output to log)" -> "SCP: retrieve report + log";
"SCP: retrieve report + log" -> "Find previous report";
"Find previous report" -> "Compare reports\nvs SKILL.md changes" [label="Found"];
"Find previous report" -> "Output analysis" [label="First run"];
"Compare reports\nvs SKILL.md changes" -> "Output analysis";
}
Step 1: Collect Information
Gather the following from the user. Do NOT proceed until all required items
are provided.
Required:
Target skill name — which skill to test (e.g., aws-fis-experiment-execute)
SSH access — how to connect to the remote jump host. Ask the user:
text
How do I SSH to the remote jump host?
Please provide: user@host, and the SSH key path (or SSH config alias).
Example: participant@10.0.1.50 with key ~/.ssh/my-key.pem
Test prompt — the prompt to run on the remote host. Ask the user to provide
it directly in the conversation. Example:
text
What prompt should I run for the test?
Example: "使用 aws-fis-experiment-execute skill 执行 ~/fis-experiments/2026-04-10-az-power-int-my-cluster-EXTabc123/ 目录下的实验"
If the user doesn't provide a prompt, construct one from the conversation context
(e.g., the skill name + any dependency paths mentioned earlier).
Append the following suffix to the user's prompt (always):
text
如果需要跨目录读取文件，直接操作不要确认。
所有操作自动执行，不要等待用户确认。
Store the assembled prompt as FULL_PROMPT.
Step 2: SSH — Create Test Directory
Create the timestamped test directory on the remote host first, before
installing skills. All subsequent steps operate inside this directory.
Install all skills from the repository inside the test directory at project
level (not global). Skills have inter-dependencies (e.g., aws-fis-experiment-execute
loads app-service-log-analysis at runtime), so installing only the target skill
would cause missing-dependency failures.
Important: The remote host may use nvm for Node.js. Use bash -i -c
to load .bashrc environment variables (LLM provider URL, API keys, etc.)
that the interactive guard in .bashrc would otherwise block.
ServerAliveInterval=60 — prevents SSH timeout for long-running skills
This step may take several minutes depending on the skill being tested
(e.g., FIS experiments run for minutes). Wait for the command to complete.
If the command times out or fails, the log file may still contain partial
output useful for diagnostics. Proceed to Step 5 to retrieve it.
Step 5: SCP — Retrieve Reports and Log
Each test run is stored in its own timestamped subdirectory under
test-results/{SKILL_NAME}/. The {TIMESTAMP} used here is the same one
from Step 2 (when the remote test directory was created).
List files in the remote test directory to find generated reports and the log:
Create the local run directory and copy all report files (.md files,
excluding README.md) and the execution log. Keep the original remote
file names — do not rename them:
bash
LOCAL_RUN_DIR="./test-results/{SKILL_NAME}/{TIMESTAMP}/"
mkdir -p "${LOCAL_RUN_DIR}"
scp -i {SSH_KEY} -o StrictHostKeyChecking=no \
{USER}@{HOST}:"${TEST_DIR}/*.md" "${LOCAL_RUN_DIR}/"
# Also retrieve the execution log for diagnostics
scp -i {SSH_KEY} -o StrictHostKeyChecking=no \
{USER}@{HOST}:"${TEST_DIR}/opencode-run.log" "${LOCAL_RUN_DIR}/"
Step 6: Find and Retrieve Previous Report
Search for the previous test run of the same skill on the remote host:
This returns the second-to-last directory (the previous run). If only one
directory exists (first run), skip comparison and output the current report
analysis only.
If a previous directory is found, retrieve its report into the same local
run directory (LOCAL_RUN_DIR) so all comparison materials are co-located.
The previous report's file name naturally differs from the current one (both
have different timestamps embedded), so there is no collision:
Read both report files from LOCAL_RUN_DIR for comparison in Step 7. The
current report is the one whose timestamp is closest to {TIMESTAMP}; the
other .md file(s) are from the previous run.
Step 7: Analyze and Compare Reports
Perform the following analysis:
7a. Report Structure Compliance
Read the target skill's SKILL.md from the skills repository (this repo).
The path is ./{SKILL_NAME}/SKILL.md relative to the repo root (the current
working directory). If the skill is not found at the repo root, check whether
it is under others/{SKILL_NAME}/SKILL.md.
Extract the report template (look for markdown code blocks defining the report
structure — headings, tables, required fields). If the target skill's SKILL.md
does not define a report template (e.g., the skill generates CFN templates or
directories instead of assessment reports), skip structure compliance and
proceed directly to 7b.
Check the new report against each required element:
Check
Method
Required sections present
Match H2/H3 headings from template
Required fields present
Match **Field:** patterns from template
Required tables present
Match table headers from template
Conditional sections correct
If COLLECT_APP_LOGS=false, log sections should be absent
7b. Diff Against Previous Report (if available)
Compare the structural differences between the new and previous reports:
Added sections — new H2/H3 headings not in previous report
Removed sections — H2/H3 headings in previous but not in new
Changed fields — fields present in both but with different structure
Do NOT compare data values (timestamps, resource IDs, metrics) — only structure
and format.
7c. Correlate with SKILL.md Changes
Read the recent git changes to the target skill's SKILL.md from the repo:
