T09 · Insecure Skill Coding Practices
- Location
scripts/make_runs.py:123- Finding
Shell Command Injection Through Evaluation Queries
- Content
View full analysis
Vulnerability Details
File Location:
scripts/make_runs.py:123-124
Vulnerability Type: OS command injection
Risk Level: HighVulnerable code:
python out = subprocess.run(a.command.replace("{query}", query), shell=True, capture_output=True, text=True, timeout=a.timeout)Technical Analysis
In
cmdmode,queryis read from an entry inevalset.jsonl, substituted directly into the user-supplied command template, and passed tosubprocess.runwithshell=True. Consequently, shell metacharacters in a query are interpreted as executable shell syntax rather than inert retrieval text.Evaluation queries may be generated from imported document corpora through an LLM. A malicious corpus provider, compromised generated evaluation set, or independent producer able to modify
evalset.jsonlcan therefore influence the query inserted into the shell command. The documented practice of placing{query}inside quotes does not prevent injection because a crafted query can terminate the quoting context.This crosses a trust boundary from evaluation data, which should be treated purely as text, into the command interpreter. The timeout only limits execution duration and does not prevent injected commands.
Attack Path
- An attacker controls or influences a document corpus used to generate the evaluation set, or directly supplies/modifies an
evalset.jsonlfile. - A resulting
querycontains shell syntax designed to escape the command template's intended argument context. - The user or agent invokes
make_runs.pywith--mode cmdand a command template containing{query}. - Line 123 performs unescaped string replacement.
shell=Truecauses the host shell to interpret the injected syntax.- The attacker's command executes under the OS identity and permissions of the Skill runner.
Impact Assessment
Successful exploitation permits arbitrary command executio ...[truncated 510 chars]
- An attacker controls or influences a document corpus used to generate the evaluation set, or directly supplies/modifies an
- Remediation
View remediation
Remediation Suggestions
- Remove
shell=Trueand execute a structured argument vector withshell=False. - Parse or define the command template as discrete arguments, then replace
{query}only within individual argument values. - Prefer passing the query to a fixed retrieval executable through standard input or a dedicated argument rather than interpolating it into command text.
- Do not rely on quote wrapping or shell escaping as the primary defense.
- Reject configurations that require shell operators, redirections, pipelines, or command substitution.
- Add regression tests using queries containing quotes, semicolons, command substitutions, newlines, and other shell metacharacters, verifying that they remain literal data.
- Remove