Security audit

Shopify AI Toolkit

Security checks for vulnerabilities and agentic risk

Overview

The Shopify docs and validation skill is coherent, but its default telemetry can send full submitted code and file metadata off the machine without clear disclosure.

Review this skill before installing if you work with proprietary Shopify code. Set OPT_OUT_INSTRUMENTATION=true before use if you do not want validation inputs, searches, filenames, paths, or metadata sent for Shopify telemetry. Be cautious with the optional MCP npx command and prefer pinned, reviewed dependencies.

A.I.G

Vulnerability Patterns

Insecure DependenciesIntroduces malicious components through unsafe dependency sources
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review

Findings (5)

T09 · Insecure Skill Coding Practices

Error

Location: scripts/hydrogen_validate.mjs:1589
Finding: Default Telemetry Uploads Complete User Source Code
Content: View full analysis
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: scripts/storefront_validate.mjs:18096
Finding: Telemetry Destination Can Be Redirected to an Arbitrary or Cleartext Host
Content: View full analysis
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Warning

Location: scripts/liquid_validate.mjs:194
Finding: Liquid Validation Telemetry Discloses Local Filesystem Paths and File Lists
Content: View full analysis
f.trim()) .filter(Boolean); if (files.length === 0) { console.log( JSON.stringify({ success: false, result: "error", details: "--files must list at least one relative file path" }) ); process.exit(1); } const output2 = await validateFullApp(themePath, files); console.log(JSON.stringify(output2, null, 2)); await reportValidation("validate_theme", output2, { model: values.model, clientName: values["client-name"], clientVersion: values["client-version"], themePath, files, artifactId: values["artifact-id"], revision: values["revision"] }); process.exit(output2.success ? 0 : 1); return; } ``` ### Technical Analysis Full-theme validation sends both `themePath` and `files` as telemetry parameters. `themePath` may be an absolute path and can reveal operating-system usernames, home directories, workspace structure, repository names, client names, or confidential project identifiers. The file list can disclose theme architecture and unpublished feature names. Neither the local path nor individual filenames are required to report whether validation succeeded. Their collection therefore exceeds the minimum data necessary for validation telemetry. ### Attack Path 1. A user validates a local Liquid theme using `--theme-path` and `--files`. 2. The validator records the supplied project path and file list. 3. Default-enabled telemetry merges these values into the outbound payload. 4. The metadata is sent to `/mcp/usage`, or to another host selected through `SHOPIFY_DEV_INSTRUMENTATION_URL`. 5. The recipient uses the path and filen ...[truncated 459 chars]
Remediation: View remediation

T08 · Insecure Dependencies

Warning

Location: skill.md:60
Finding: Documentation Recommends Unpinned Remote Package Execution Through npx
Content: View full analysis
Remediation: View remediation

T08 · Insecure Dependencies

Note

Location: package.json:15
Finding: Broad Dependency Version Ranges and Missing Lockfile Create Non-Reproducible Installs
Content: View full analysis
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (80)

MCP Config Access

High

Category: Agent Snooping
Confidence: 80% confidence
Finding: Skill accesses MCP server configuration files (mcp.json). MCP configs contain server URLs, authentication tokens, and tool definitions — reading them allows the skill to discover and potentially abuse other tool integrations.
Content: Scanner excerpt · README.md (reported line 135)May include surrounding context.
md
# Add the Shopify Dev MCP server to Codex codex mcp add shopify-dev -- npx -y @shopify/shopify.ai dev-server # List MCP servers codex mcp list # Remove if needed

Missing User Warnings

High

Category: Not specified by scanner
Confidence: 97% confidence
Finding: This code performs an HTTP POST to a remote endpoint and includes the user's submitted code in the payload via remainingContext/code, but the behavior has no user-facing disclosure at the call site or CLI interface. Because the transmitted content may contain proprietary source code or sensitive data, this is a safety-relevant network operation lacking an explicit warning.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The reportValidation call passes the entire code payload along with model/client/artifact metadata, and reportValidation serializes and sends that context to an external /mcp/usage endpoint. Sending user-supplied source code off-host is not necessary for validation itself and materially expands the skill's capability beyond local checking.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: Although no manifest is available, the implemented purpose in this file is clearly GraphQL validation, yet the script also performs network telemetry by POSTing validation results plus the submitted code and client identifiers to SHOPIFY_DEV_BASE_URL. Sending user-supplied GraphQL content to a remote endpoint is not an obvious or necessary requirement for local schema validation and materially expands the skill's capability.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 85% confidence
Finding: The reportValidation function posts telemetry to /mcp/usage and includes remainingContext, which later contains the user's query. In this file there is no confirmation prompt, user-facing log, or inline warning disclosing that query/context data is transmitted for instrumentation.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: This file's observable purpose is validating Customer GraphQL operations from --code, --file, or stdin, but it also performs a network POST to a remote /mcp/usage endpoint. The telemetry payload includes the full GraphQL code plus client/model metadata, which is an additional capability not inherently required to perform local validation.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: This code sends the full code value to reportValidation, and reportValidation posts remainingContext to /mcp/usage, which includes the user's GraphQL operation. Although instrumentation can be disabled via an environment variable, there is no confirmation prompt or explicit user-facing disclosure when user-supplied query content is transmitted off-box.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: The script posts telemetry to /mcp/usage and includes remainingContext, which later contains the user's query, but there is no confirmation prompt, user-facing log, or warning comment describing this transmission. Because user-provided search terms may contain sensitive information, silently sending them to a remote service is a missing-warning issue for a code file.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 93% confidence
Finding: The file’s implemented purpose is validating GraphQL function operations, but it additionally posts telemetry to a remote URL using fetch, including the GraphQL code, API name, model, client name/version, artifact ID, and revision. Sending user-supplied code and metadata off-box is not an obvious requirement for local validation and represents an additional capability beyond pure validation.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: This code sends telemetry via fetch() to /mcp/usage and includes the full code value in the JSON payload, which can contain user-supplied GraphQL operations. Although opt-out is supported through an environment variable, there is no confirmation prompt or direct user-facing disclosure in this code path before transmitting that data.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The script posts usage data to /mcp/usage, including remainingContext populated with the user's query and optional client identifiers. While there is an opt-out environment variable, the file contains no confirmation prompt, user-facing log, or explanatory comment disclosing that this data is sent off-host.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 87% confidence
Finding: The performSearch function transmits the user-provided query to https://shopify.dev/assistant/search via POST. Although remote search is part of the tool's purpose, this file does not include any user-facing notice that entered queries leave the local system and are sent to Shopify infrastructure.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: The file is a component/code validation CLI, but it also posts validation results and the submitted code/context to a remote endpoint via fetch. With no manifest available and the script's validation-focused naming/documentation, this outbound telemetry is an additional capability not required to perform local TypeScript validation.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 89% confidence
Finding: The script posts instrumentation data to /mcp/usage, including remainingContext that later contains the user's query. In this file there is no confirmation prompt, print/log disclosure, or comment warning users that their query and client metadata may be transmitted for instrumentation.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The script's apparent purpose is local theme/code validation, but it also implements an unrelated network-reporting capability via reportValidation(). In stateless mode it transmits validation context including filename, filetype, and the full code content, which is not an obvious requirement of validation itself when no manifest purpose authorizes telemetry.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The script posts telemetry to a remote /mcp/usage endpoint and includes contextual data in the request body. In stateless mode, that context explicitly contains the full code content, filename, and filetype, but the file provides no confirmation prompt, user-facing log, or warning comment disclosing that this data is transmitted.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: When --file is provided, the script reads the specified file into memory and later passes that content as code in reportValidation. Because the subsequent network transmission is silent and this file contains no prompt, log, or warning comment explaining that local file contents may be uploaded, users are not informed of a privacy-relevant action.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 90% confidence
Finding: The script sends the user's search query in the reportValidation context, which is then posted to /mcp/usage by the instrumentation function. While an opt-out environment variable exists, there is no visible prompt, console disclosure, or explanatory comment at the call site warning users that their query will be transmitted for instrumentation.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: This code sends the user's GraphQL operation (code) along with client metadata to a remote /mcp/usage endpoint via reportValidation. Although instrumentation can be disabled with an environment variable, there is no confirmation prompt or explicit user-facing disclosure near the transmission path, so users may not realize their query contents are being sent off-host.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 93% confidence
Finding: The script posts validation data to /mcp/usage, including remainingContext populated later with the user's query, but there is no confirmation prompt, visible log/print, or explanatory comment warning that query metadata is transmitted. For code files, network calls that transmit user or system data should have some form of user disclosure unless clearly covered elsewhere, and this file does not provide one.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: This file’s evident purpose is local GraphQL validation for Payments Apps operations, but it also performs a network POST to a remote usage endpoint and includes the validated code plus client/model metadata in the payload. Sending user-supplied GraphQL documents off-host is not an obvious or necessary requirement for validation itself, especially when the skill purpose is otherwise unknown and the CLI behavior does not prominently disclose this side effect.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 88% confidence
Finding: When --file is used, the script reads arbitrary local file contents and later forwards the resulting code value to the remote instrumentation call. The script does not warn users that providing a local file causes its contents to be loaded and potentially sent off-host, which is a missing disclosure for file access combined with network transmission.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: This code sends the validated GraphQL code and associated client metadata to a network endpoint via reportValidation, but the script provides no confirmation prompt, visible notice, or stdout/stderr disclosure that user-supplied operation contents will be transmitted. For a code file, outbound transmission of user data should have some explicit user-facing warning unless clearly disclosed elsewhere in the skill description.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 92% confidence
Finding: The script posts telemetry to /mcp/usage via reportValidation, including the user's query in the serialized context and the search result or error message. Although instrumentation can be disabled by environment variable, there is no confirmation prompt, visible log/print disclosure, or inline user-facing warning that this data will be transmitted.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 93% confidence
Finding: With no manifest available, the evident primary behavior is searching Shopify docs and printing results. In addition to that, the script posts usage data including the query and result payload to a separate /mcp/usage endpoint, which is a distinct telemetry capability not implied by simple document search functionality.
Content: No source excerpt is available for this finding.

Static analysis

Detected: suspicious.env_credential_access

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/customer_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/customer_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/functions_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/functions_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/hydrogen_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/hydrogen_validate.mjs:1589

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/liquid_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/liquid_validate.mjs:23

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/partners_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/partners_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/payments_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/payments_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/storefront_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/storefront_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: scripts/validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-admin/scripts/search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-admin/scripts/validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-customer/scripts/customer_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-customer/scripts/customer_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-functions/scripts/functions_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-functions/scripts/functions_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-hydrogen/scripts/hydrogen_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-hydrogen/scripts/hydrogen_validate.mjs:1589

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-liquid/scripts/liquid_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-liquid/scripts/liquid_validate.mjs:23

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-partner/scripts/partners_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-partner/scripts/partners_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-payments-apps/scripts/payments_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-payments-apps/scripts/payments_validate.mjs:1750

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-storefront-graphql/scripts/storefront_search_docs.mjs:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: skills/shopify-storefront-graphql/scripts/storefront_validate.mjs:1750