Back to skill

Security audit

experiment-framework

Security checks for vulnerabilities and agentic risk

Overview

This skill is a coherent ML experiment runner, but it has broad credential exposure and an unsafe results path that users should review before installing.

Install only in an isolated environment and run only trusted experiment specs and source commits. Avoid the optional unpinned npx installer, use narrowly scoped temporary tokens, review bundles before publishing, avoid --repo-dir publishing until results_prefix containment is fixed, and do not run untrusted experiment commands with credentials in your shell environment.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
src/axiotic_experiments/github_store.py:40
Finding
Unvalidated results prefix permits filesystem path traversal, deletion, and overwrite<![CDATA[ ## Vulnerability Details **File Location**: `src/axiotic_experiments/spec.py:340`; `src/axiotic_experiments/github_store.py:40-55` **Vulnerability Type**: Path traversal and unsafe recursive deletion **Risk Level**: High ### Vulnerable Code ```python # src/axiotic_experiments/spec.py github=GitHubSpec( repo=str(_required(github, "repo", "github")), results_branch=str(_required(github, "results_branch", "github")), results_prefix=str(github.get("results_prefix", "experiments/results")), ), ``` ```python # src/axiotic_experiments/github_store.py class GitHubResultStore: def __init__(self, *, repo_dir: Path, results_prefix: str): self.repo_dir = repo_dir.expanduser().absolute() self.results_prefix = results_prefix.strip("/") def stage(self, bundle_dir: Path, *, run_id: str) -> Path: assert_no_secret_material(bundle_dir) destination = self.repo_dir / self.results_prefix / run_id destination.mkdir(parents=True, exist_ok=True) for source in bundle_dir.iterdir(): target = destination / source.name if source.is_dir(): if target.exists(): shutil.rmtree(target) shutil.copytree(source, target) elif source.is_file(): shutil.copy2(source, target) return destination ``` ### Technical Analysis The specification parser accepts `[github].results_prefix` without rejecting absolute paths or `..` path components. Calling `strip("/")` removes leading and trailing separators but does not neutralize traversal sequences. `GitHubResultStore.stage()` joins this attacker-controlled value to `repo_dir` and then performs directory creation, recursive deletion, and file copying. A value such as `../../victim` can cause the normalized destination to escape the intended repository. The recursive `shutil.rmtree(target)` operation makes this more severe than a simple arbitrary write: an existing direc ...[truncated 968 chars]
Remediation
<![CDATA[ ## Remediation Suggestions - Reject absolute `results_prefix` values and any path containing `..` components during specification validation. - Resolve the final destination before performing filesystem operations and verify that it remains under `repo_dir`: ```python base = self.repo_dir.resolve() destination = (base / self.results_prefix / run_id).resolve() if not destination.is_relative_to(base): raise GitHubStoreError("results destination escapes repository root") ``` - Apply the same containment check to every target before deletion or copying. - Validate `results_prefix` against a conservative repository-relative path format. - Add regression tests covering `../`, nested traversal, absolute paths, repeated separators, and symlink-assisted escapes. - Avoid recursively deleting an existing target unless it has first been proven to reside under the expected results directory. ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
src/axiotic_experiments/github_store.py:19
Finding
Secret scanning can be bypassed before files are uploaded to GitHub<![CDATA[ ## Vulnerability Details **File Location**: `src/axiotic_experiments/github_store.py:19-36, 131-145` **Vulnerability Type**: Incomplete secret detection before remote publication **Risk Level**: Medium ### Vulnerable Code ```python _TOKEN_PATTERNS = ( re.compile(r"\brpa_[A-Za-z0-9_-]{20,}\b"), re.compile(r"\bgh[opsu]_[A-Za-z0-9]{20,}\b"), re.compile(r"\bhf_[A-Za-z0-9]{20,}\b"), re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b"), ) def assert_no_secret_material(directory: Path) -> None: for path in directory.rglob("*"): if not path.is_file() or path.stat().st_size > 5_000_000: continue try: text = path.read_text(encoding="utf-8") except UnicodeDecodeError: continue if any(pattern.search(text) for pattern in _TOKEN_PATTERNS): raise GitHubStoreError(f"credential-like material detected in {path}") ``` ```python def upload_bundle(self, bundle_dir: Path, *, run_id: str) -> str: assert_no_secret_material(bundle_dir) files = sorted(path for path in bundle_dir.iterdir() if path.is_file()) if not files: raise GitHubStoreError(f"result bundle is empty: {bundle_dir}") commit = "" for path in files: commit = self.upload_file(path, run_id=run_id) return commit def upload_file(self, source: Path, *, run_id: str) -> str: path = f"{self.results_prefix}/{run_id}/{source.name}" existing = self._request_json("GET", path, missing_ok=True) payload: dict[str, Any] = { "message": f"[{run_id}] publish {source.name}", "content": base64.b64encode(source.read_bytes()).decode(), "branch": self.branch, } ``` ### Technical Analysis The scanner deliberately skips files larger than 5 MB and files that are not valid UTF-8. Those same files remain eligible for upload. Its pattern set also recognizes only a small number of credential formats and does not detect generic password assignments, W&B keys, private ...[truncated 1198 chars]
Remediation
<![CDATA[ ## Remediation Suggestions - Fail closed for files that cannot be inspected rather than silently skipping them. - Enforce an explicit allowlist of result file types and maximum publication sizes. - Either reject binary files by default or scan their extracted/textual content using an appropriate secret-scanning engine. - Expand detection to cover private keys, password assignments, W&B credentials, generic high-entropy tokens, and additional provider formats. - Reject or carefully resolve symlinks before scanning and publication. - Ensure the exact bytes approved by the scanner are the bytes uploaded, avoiding time-of-check/time-of-use replacement. - Display a publication manifest and require explicit authorization for unrecognized or excluded file types. - Add tests proving that large files, binary files, symlinks, generic credentials, and W&B keys cannot bypass the publication gate. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
src/axiotic_experiments/backends.py:188
Finding
Experiment commands and SSH targets receive all configured service credentials<![CDATA[ ## Vulnerability Details **File Location**: `src/axiotic_experiments/credentials.py:14-38`; `src/axiotic_experiments/backends.py:188-229` **Vulnerability Type**: Excessive credential exposure and violation of least privilege **Risk Level**: Medium ### Vulnerable Code ```python # src/axiotic_experiments/credentials.py BASE_CREDENTIALS = ("WANDB_API_KEY", "HF_TOKEN", "GITHUB_TOKEN") RUNPOD_CREDENTIALS = ("RUNPOD_API_KEY",) def required_credentials(profile: MachineProfile) -> tuple[str, ...]: if profile.transport == "runpod": return BASE_CREDENTIALS + RUNPOD_CREDENTIALS return BASE_CREDENTIALS def resolve_credentials( profile: MachineProfile, environ: Mapping[str, str] | None = None, ) -> dict[str, str]: source = environ if environ is not None else os.environ names = required_credentials(profile) missing = [name for name in names if not source.get(name)] if missing: raise CredentialError( "missing runtime credential: " + ", ".join(sorted(missing)) ) return {name: source[name] for name in names} ``` ```python # src/axiotic_experiments/backends.py def launch_local( spec: ExperimentSpec, profile: MachineProfile, *, runtime_env: dict[str, str], credentials: dict[str, str], ) -> LaunchResult: workdir = _prepare_local_source(spec, profile) env = os.environ.copy() env.update(runtime_env) env.update(credentials) started = time.monotonic() process = subprocess.Popen( list(spec.run.command), cwd=workdir, env=env, start_new_session=True, ) ``` ```python def ssh_payload( spec: ExperimentSpec, profile: MachineProfile, *, runtime_env: dict[str, str], credentials: dict[str, str], ) -> dict[str, Any]: return { "run_id": spec.run.id, "command": list(spec.run.command), "env": {**runtime_env, **credentials}, "timeout_seconds": spec.execution.max_hours * 3600, ...[truncated 2279 chars]
Remediation
<![CDATA[ ## Remediation Suggestions - Resolve credentials per operation rather than requiring one universal credential set. - Do not provide `GITHUB_TOKEN` to trainer processes; retain it in the dispatcher that publishes final results. - Provide Hugging Face credentials only to commands explicitly configured to upload checkpoints or datasets. - Use short-lived, narrowly scoped tokens restricted to the exact repository and required actions. - Separate read-only and write credentials and default to the minimum scope. - For SSH execution, use host-specific temporary credentials or a brokered upload mechanism instead of forwarding long-lived tokens. - Require an explicit declaration of required integrations in the specification and display the resulting credential grants before execution. - Consider isolated subprocesses or helper services that perform authenticated operations without exposing raw tokens to experiment code. ]]>

T08 · Insecure Dependencies

Warning
Location
README.md:15
Finding
Optional installation command relies on unpinned mutable remote tooling<![CDATA[ ## Vulnerability Details **File Location**: `README.md:15` **Vulnerability Type**: Unpinned remote installer and mutable supply-chain source **Risk Level**: Medium ### Vulnerable Code ```sh npx skills add AntreasAntoniou/experiment-framework ``` ### Technical Analysis The documented optional installation path invokes `npx` without pinning the package version used to provide the `skills` command. It also identifies the Skill by a mutable repository reference rather than a reviewed commit SHA or integrity-verified artifact. Consequently, code or content obtained when the command is run may differ from the version covered by this audit. A compromised npm package, maintainer account, repository, tag, or default branch could change installation behavior after review. This is a supply-chain weakness rather than evidence that the currently reviewed repository contains a malicious installer. ### Attack Path 1. A user follows the documented optional `npx` installation command. 2. `npx` resolves the current available implementation of the unpinned `skills` package. 3. The installer resolves the current mutable state of the referenced remote Skill repository. 4. An upstream compromise or later malicious modification causes different code or instructions to be downloaded or executed. 5. That code runs or is installed with the invoking user's privileges and may subsequently be loaded by the agent. ### Impact Assessment A successful upstream compromise could execute code with the user's privileges during installation or install altered Skill instructions and scripts into an agent environment. The practical scope includes files, credentials, and agent configuration accessible to the installing user. ]]>
Remediation
<![CDATA[ ## Remediation Suggestions - Pin the npm installer to an audited version, for example by using an explicit package version. - Pin the Skill repository to a full immutable commit SHA rather than a branch or mutable default reference. - Publish and verify cryptographic checksums or signed release artifacts. - Prefer the documented local isolated Python installation for security-sensitive environments. - Explain that `npx` may download and execute third-party code and require users to review the resolved package before execution. - Adopt dependency lock files and automated provenance verification for release artifacts. ]]>
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (58)

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
90% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
Hugging Face uploads and GitHub/Hugging Face validation behaviors materially expand the attack surface beyond local experiment management, especially if artifacts can contain checkpoints, dataset fragments, or embedded secrets. In this context, undeclared publication to public or misconfigured private repositories could result in irreversible data exposure.

Env Variable Harvesting

High
Category
Data Exfiltration
Content
credentials: dict[str, str],
) -> LaunchResult:
    workdir = _prepare_local_source(spec, profile)
    env = os.environ.copy()
    env.update(runtime_env)
    env.update(credentials)
    started = time.monotonic()
Confidence
96% confidence
Finding
Copying the full process environment into a job that then executes attacker-controlled or user-controlled experiment commands is a classic secret overexposure problem. Any tokens present in the launcher's environment can become accessible to the experiment process, which is especially dangerous in a framework designed to run arbitrary ML workloads and remote code from repositories.

Credential Access

High
Category
Privilege Escalation
Content
isinstance(key, str) and isinstance(value, (str, int, float, bool))
            for key, value in public_env_raw.items()
        ):
            raise SpecError("[execution].env must contain scalar public values")
        public_env = {key: str(value) for key, value in public_env_raw.items()}
        for key, value in public_env.items():
            if _SECRET_KEY.search(key.upper()):
Confidence
60% confidence
Finding
Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Credential Access

High
Category
Privilege Escalation
Content
isinstance(key, str) and isinstance(value, (str, int, float, bool))
            for key, value in public_env_raw.items()
        ):
            raise SpecError("[execution].env must contain scalar public values")
        public_env = {key: str(value) for key, value in public_env_raw.items()}
        for key, value in public_env.items():
            if _SECRET_KEY.search(key.upper()):
Confidence
60% confidence
Finding
Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Credential Access

High
Category
Privilege Escalation
Content
isinstance(key, str) and isinstance(value, (str, int, float, bool))
            for key, value in public_env_raw.items()
        ):
            raise SpecError("[execution].env must contain scalar public values")
        public_env = {key: str(value) for key, value in public_env_raw.items()}
        for key, value in public_env.items():
            if _SECRET_KEY.search(key.upper()):
Confidence
60% confidence
Finding
Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Rp1

Medium
Category
MCP Rug Pull
Confidence
89% confidence
Finding
The README instructs users to install the skill via `npx skills add AntreasAntoniou/experiment-framework` without pinning a specific version or commit. This can lead to non-reproducible installs and creates a supply-chain risk where a later compromised or maliciously updated package/version is fetched and executed in the user's environment.

Lp3

Medium
Category
MCP Least Privilege
Confidence
93% confidence
Finding
The skill metadata declares no explicit tool scope or permissions even though the skill description clearly implies shell execution, filesystem access, environment-variable use, and likely networked companion tooling. Missing scope declarations are dangerous because reviewers and calling agents cannot reliably constrain what the skill may access, increasing the chance of unauthorized file, secret, or network operations.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
def _git_head(path: Path) -> str:
    completed = subprocess.run(
        ["git", "-C", str(path), "rev-parse", "HEAD"],
        check=False,
        capture_output=True,
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
workdir = Path(profile.remote_checkout(spec.run.id)).expanduser()
        if not (workdir / ".git").is_dir():
            workdir.parent.mkdir(parents=True, exist_ok=True)
            subprocess.run(
                ["git", "clone", "--no-checkout", spec.source.clone_url, str(workdir)],
                check=True,
            )
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
["git", "clone", "--no-checkout", spec.source.clone_url, str(workdir)],
                check=True,
            )
        subprocess.run(
            [
                "git",
                "-C",
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
],
            check=True,
        )
        subprocess.run(
            ["git", "-C", str(workdir), "checkout", "--detach", spec.source.commit],
            check=True,
        )
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Static analysis

No suspicious patterns detected.