Back to skill

Security audit

dataset-producer

Security checks for vulnerabilities and agentic risk

Overview

This is a coherent dataset-packaging skill, but it needs review because its scripts can expose sensitive data in logs and can leave old dataset shards in publishable output folders.

Install only if you are comfortable reviewing the scripts before use. Avoid running validation on confidential or regulated data unless logs are controlled, mask or disable raw PII output, build into a fresh output directory for every run, and inspect the dataset folder before any HuggingFace upload.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/validate.py:464
Finding
PII Validator Discloses Sensitive Values in Plaintext Logs<![CDATA[ ## Vulnerability Details **File Location**: `scripts/validate.py`, lines 226–250 and 450–465 **Vulnerability Type**: Plaintext sensitive-data exposure **Risk Level**: Medium **Category**: T09: Insecure Skill Coding Practices ### Vulnerable Code ```python def scan_pii(rows: List[Dict[str, Any]], include_contextual: bool = False) -> Dict[str, List[Tuple[int, str, str]]]: """Scan all string fields for PII patterns. Returns findings dict.""" patterns = dict(DEFAULT_PII_PATTERNS) if include_contextual: patterns.update(CONTEXTUAL_PII_PATTERNS) findings: Dict[str, List[Tuple[int, str, str]]] = { k: [] for k in patterns } for i, row in enumerate(rows): for field, val in row.items(): if not isinstance(val, str): continue for pii_type, pattern in patterns.items(): for m in pattern.finditer(val): if pii_type == "credit_card" and not _luhn_ok(m.group()): continue findings[pii_type].append((i, field, m.group())) break return findings ``` The collected sensitive value is subsequently printed without masking: ```python for row_idx, field, match in hits[:3]: print(f" row {row_idx}.{field}: {match!r}") ``` ### Technical Analysis The PII scanner collects the complete regex match through `m.group()`. This may contain an email address, telephone number, Social Security number, or Luhn-valid payment-card number. The reporting logic then writes up to three complete values from each finding category to standard output. A validation report does not need the complete sensitive value to identify the affected record. The row index, field name, PII type, and a masked suffix are sufficient for remediation. Printing the complete value violates data minimization and expands the number of systems holding the sensitive information. Although the script does not di ...[truncated 1872 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Do not store or print complete PII matches by default. 2. Change the findings structure to retain only: - Row index - Field name - PII category - Optional masked preview 3. Apply category-specific masking, for example: - SSN: `***-**-1234` - Card number: `************1234` - Email: `a***@example.com` - Telephone: `***-***-1234` 4. Prefer count-only output for automated and CI environments. 5. If raw-value inspection is genuinely required, place it behind an explicit option such as `--show-raw-pii`, accompanied by a prominent warning. 6. Send any explicitly requested raw report to a restricted local file rather than standard output, using owner-only permissions where supported. 7. Add tests asserting that standard output never contains known fixture SSNs, card numbers, emails, or telephone numbers. 8. Document that validation logs may contain sensitive metadata such as row indices and field names even after values are masked. A safer reporting pattern would be: ```python def mask_match(pii_type: str, value: str) -> str: if pii_type in {"us_ssn", "credit_card", "us_phone"}: digits = re.sub(r"\D", "", value) return f"***{digits[-4:]}" if len(digits) >= 4 else "***" if pii_type == "email": local, _, domain = value.partition("@") return f"{local[:1]}***@{domain}" if domain else "***" return "***" for row_idx, field, match in hits[:3]: print( f" row {row_idx}.{field}: " f"{mask_match(pii_type, match)!r}" ) ``` ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/produce_dataset.py:563
Finding
Dataset Rebuilds Can Retain and Publish Stale Parquet Shards<![CDATA[ ## Vulnerability Details **File Location**: `scripts/produce_dataset.py`, lines 563–590 and 685–686 **Vulnerability Type**: Stale sensitive-data retention in reused output directories **Risk Level**: Medium **Category**: T09: Insecure Skill Coding Practices ### Vulnerable Code The pipeline creates or reuses the output directory without checking whether it contains shards from a prior build: ```python out_dir = Path(out_dir) out_dir.mkdir(parents=True, exist_ok=True) shard_size_bytes = parse_size(shard_size) ``` It then writes only the shards needed by the current build: ```python def write_shards(rows: List[Dict[str, Any]], features: Features, split_name: str, out_dir: Path, shard_size_bytes: int, quiet: bool = False) -> Tuple[List[Path], int]: """Write rows as sharded parquet for one split.""" shards = shard_rows(rows, shard_size_bytes) n_shards = len(shards) paths: List[Path] = [] total_nbytes = 0 for i, shard_rows_list in enumerate(shards): ds = _rows_to_dataset(shard_rows_list, features) fname = f"{split_name}-{i:05d}-of-{n_shards:05d}.parquet" fpath = out_dir / fname if quiet: import contextlib import io with contextlib.redirect_stderr(io.StringIO()), \ contextlib.redirect_stdout(io.StringIO()): ds.to_parquet(str(fpath)) else: ds.to_parquet(str(fpath)) paths.append(fpath) return paths, total_nbytes ``` There is no cleanup, manifest comparison, non-empty-directory rejection, or atomic replacement before current shards are written. ### Technical Analysis Shard names include the current total shard count. If an earlier build generated more shards, a later smaller build writes new filenames but does not remove the old files. The same problem occurs when split names change or a split is removed. For example: ```text First build: train-00000-of-0000 ...[truncated 2443 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Default to refusing a non-empty output directory. 2. Require an explicit `--overwrite` option before replacing an existing build. 3. When overwrite is authorized, remove only files managed by the pipeline: - Validate split names before using them in patterns. - Delete matching Parquet shards and stale `stats.json`. - Do not recursively delete arbitrary user files. 4. Prefer building into a newly created temporary sibling directory. 5. Validate the completed temporary build and then atomically replace the previous output directory. 6. Generate a manifest containing every expected shard name and checksum. 7. Before publication, fail if any Parquet file in the output directory is absent from the current manifest. 8. Modify upload guidance to upload only manifest-listed artifacts or a freshly generated staging directory. 9. Add regression tests covering: - Rebuilding from three shards to one - Removing a split - Renaming a split - Rebuilding after PII redaction - Ensuring no obsolete shard survives A minimal guarded approach would be: ```python def prepare_output_directory(out_dir: Path, overwrite: bool) -> None: out_dir.mkdir(parents=True, exist_ok=True) existing_shards = list(out_dir.glob("*.parquet")) if existing_shards and not overwrite: raise FileExistsError( "Output directory contains existing Parquet shards; " "use --overwrite or choose a fresh directory." ) if overwrite: for shard in existing_shards: shard.unlink() ``` For stronger safety, write the full build to a temporary directory and replace the destination only after validation succeeds. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (15)

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding
The code substantially matches part of the declared purpose: it does build HuggingFace-compatible parquet shards, handles schema/features parsing, split strategies, deduplication, sharding, and statistics. However, the declared description claims a complete, publish-ready dataset pipeline including README.md with YAML frontmatter, dataset card generation, and provenance script creation. None of those capabilities are present in this code chunk. The script writes parquet data and stats.json only. Therefore the description overstates the implemented functionality in a material way, making this a mismatch.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding
The description presents a full dataset production pipeline, but the supplied code chunk only validates raw input data before sharding. Its own docstring explicitly says it is a 'Standalone validation suite' to use before running produce_dataset.py. While validation is one sub-capability mentioned in the description, this code does not implement the core claimed behavior of assembling and publishing a complete HuggingFace-format dataset. Therefore the code’s primary purpose is materially narrower and different from the declared purpose.

Anti-Refusal Statement

High
Category
Anti-Refusal
Content
for w in warnings:
            print(f"    - {w}")
    else:
        print("  ✓ No warnings")
    if critical_failure:
        print("\n  ✗ Critical failure — fix before proceeding to produce_dataset.py")
        sys.exit(1)
Confidence
80% confidence
Finding
Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Lp3

Medium
Category
MCP Least Privilege
Confidence
91% confidence
Finding
The skill explicitly instructs the agent to read local files, write dataset artifacts, and run shell commands, but it declares no tool scope or permissions boundary. In an agent framework, that creates an authorization gap where a broadly-triggered skill can access or modify the filesystem or execute commands without an explicit least-privilege contract.

Vague Triggers

Medium
Confidence
95% confidence
Finding
The trigger phrases are extremely broad (`create`, `build`, `package`, `publish`, `make a dataset`) and can cause the skill to activate in contexts where the user did not intend a filesystem-writing, shell-capable workflow. When combined with undeclared tool scope, over-triggering increases the chance of unintended data access, file creation, or execution of processing pipelines on sensitive inputs.

Missing User Warnings

Medium
Confidence
86% confidence
Finding
This skill is designed to assemble and potentially publish datasets, yet the top-level instructions do not prominently warn against processing or publishing sensitive, regulated, or personal data without explicit review and consent. In this context, omission matters because the workflow includes collection, packaging, and optional hub publication, which can turn accidental inclusion of PII or confidential data into an irreversible disclosure.

Scope Creep

Low
Category
Excessive Agency
Content
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
Confidence
70% confidence
Finding
Skill's behavior or capabilities extend beyond its stated purpose. Scope creep allows an agent to perform actions unrelated to its documented functionality, increasing the attack surface.

Natural-Language Policy Violations

Low
Confidence
89% confidence
Finding
Line L051 instructs that the `language:` field must be a list of ISO 639-1 codes, which imposes a specific language/locale representation as a blanket requirement. This is a natural-language locale policy constraint with no opt-in, exception handling, or justification for datasets that may require other locale conventions.

Unverifiable Dependency: datasets has 1 known advisory(ies) (CVE-2026-66007 (Datasets through 5.0.0, fixed in commit f989ef9, contains a path traversal vulne)), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding
Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Unpinned Dependencies

Low
Category
Supply Chain
Content
# PyArrow — parquet backend used by datasets. Pinned to a recent version
# because the YAML `dataset_size` computation depends on
# `Dataset.data.table.nbytes` which behaves consistently across 14+.
pyarrow>=14.0

# PyYAML — used by make_card.py to parse and validate the YAML frontmatter
# in the generated README.md.
Confidence
50% confidence
Finding
Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Unverifiable Dependency: pyarrow has 9 known advisory(ies) (CVE-2023-47248 (PyArrow: Arbitrary code execution when loading a malicious data file); CVE-2019-12408 (Missing Initialization of Resource in Apache Arrow); CVE-2019-12410 (Missing Initialization of Resource in Apache Arrow) +6 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding
Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Unpinned Dependencies

Low
Category
Supply Chain
Content
# PyYAML — used by make_card.py to parse and validate the YAML frontmatter
# in the generated README.md.
pyyaml>=6.0

# Optional (not installed by default):
#
Confidence
50% confidence
Finding
Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Unverifiable Dependency: pyyaml has 8 known advisory(ies) (CVE-2019-20477 (Deserialization of Untrusted Data in PyYAML); CVE-2020-1747 (Improper Input Validation in PyYAML); CVE-2020-14343 (Improper Input Validation in PyYAML) +5 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding
Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Natural-Language Policy Violations

Low
Confidence
92% confidence
Finding
The file sets `LANGUAGES = ["en"]`, which is a natural-language locale constraint embedded in code. The policy says to flag language or locale restrictions unless the skill offers user opt-in or clearly documents a justified region- or language-specific limitation, which this file does not.

Missing User Warnings

Low
Confidence
87% confidence
Finding
The script calls datasets.load_dataset() on a user-supplied local path without constraining behavior. In the Hugging Face ecosystem, dataset loading can invoke builder logic, parse repository metadata, and potentially perform network access or execute repository-provided loading code depending on dataset structure and library settings. Because this is a smoke-test utility intended to be run on arbitrary produced datasets, that makes the operation riskier than a purely local file validation step.

Static analysis

No suspicious patterns detected.