alibabacloud-agentloop-evaluation

• Outcome: Review
• Latest audit: Jul 30, 2026 · 1:03 PM UTC
• Version: 0.0.2

Overview

The skill mostly matches its stated evaluation purpose, but it needs Review because setup directs broad Aliyun CLI/plugin updates and auto-install behavior that can change the local toolchain beyond the narrow evaluation workflow.

Install only if you are comfortable with the skill using your Aliyun CLI profile to create and inspect AgentLoop evaluation resources and query SLS logs. Review the setup commands first; prefer manually installing or updating only the needed AgentLoop and SLS plugins in a test environment, avoid broad plugin updates where change control matters, run dry-runs before execution, and store output JSON files in a private path if results may contain operational or customer data.

SkillSpector

Vulnerability Patterns

• Rogue Agent: Self-Modification, Session Persistence
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration

Findings (6)

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 89% confidence
• Finding: The skill clearly instructs use of shell execution, environment variables, and file reads/writes, but no explicit permission declaration is described. In an agent framework, this mismatch can bypass least-privilege expectations and cause the host to grant broader capabilities than users or reviewers realize.

Missing User Warnings

Medium

• Confidence: 88% confidence
• Finding: The document lists destructive commands such as delete and terminate operations without any adjacent warning, confirmation guidance, or stronger emphasis on safer preview paths. In an agent skill context, this can normalize or encourage direct execution of irreversible actions, increasing the chance an agent or user issues account-impacting mutations without adequate review.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: This verification guide includes a command that performs a live evaluation task creation when `--execute` is supplied and writes results to `/tmp/eval-result.json`, but it does not explicitly warn the user that this step causes real remote side effects and local artifact creation. In a skill specifically designed to orchestrate AgentLoop evaluations, such behavior is expected, but the lack of a clear side-effect warning can still lead to unintended task creation, resource consumption, or accidental disclosure of evaluation data on shared systems.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: The analysis execution step writes structured evaluation analysis to `/tmp/eval-analysis.json` without clearly warning that potentially sensitive operational or evaluation data will be stored locally. Although this is aligned with the skill's purpose, users may run it on shared hosts or CI environments where temporary files are accessible longer than expected.

Self-Modification

High

• Category: Rogue Agent

Content

**Pre-check: Aliyun CLI >= 3.3.3 required**
> [MUST] Verify: `aliyun version` — must be >= 3.3.3.
> - **First install or major upgrade:** Download, review, then execute the [setup script](references/cli-installation-guide.md#first-time-install-or-major-upgrade). Avoid `curl | bash` piping.
> - **Routine update (CLI >= 3.3.5):** `aliyun upgrade` — prefer this built-in self-update over re-running the install script.
> - See [references/cli-installation-guide.md](references/cli-installation-guide.md) for full installation instructions.

**Pre-check: Aliyun CLI plugin update required**

• Confidence: 84% confidence
• Finding: The skill instructs operators to run self-update commands such as `aliyun upgrade` and `aliyun plugin update`, which modify local tooling outside the task's core workflow. Self-modifying or self-updating behavior increases supply-chain and integrity risk, especially in an agent-driven environment where updates may occur without normal administrative review.

Self-Modification

High

• Category: Rogue Agent

Content

| Product | CLI Command | Description |
|---------|-------------|-------------|
| CLI | `aliyun version` | Check CLI version (>= 3.3.3 required) |
| CLI | `aliyun upgrade` | Self-update CLI (>= 3.3.5) |
| CLI | `aliyun plugin show --name aliyun-cli-agentloop` | Verify AgentLoop plugin installed |
| CLI | `aliyun plugin show --name aliyun-cli-sls` | Verify SLS plugin installed |
| CLI | `aliyun plugin update` | Update all installed plugins |

• Confidence: 80% confidence
• Finding: Including `aliyun upgrade` as a routine related command introduces a self-modification capability that changes the local CLI environment. In an agent-operated workflow, recommending or invoking upgrades can alter behavior, install new code, or affect reproducibility and trust boundaries, especially if done automatically or without explicit user consent.

Static analysis

No suspicious patterns detected.