Back to skill

Security audit

dataset-producer

Security checks for vulnerabilities and agentic risk

Overview

This is a coherent dataset-packaging skill, but it needs review because its scripts can expose sensitive data in logs and can leave old dataset shards in publishable output folders.

Install only if you are comfortable reviewing the scripts before use. Avoid running validation on confidential or regulated data unless logs are controlled, mask or disable raw PII output, build into a fresh output directory for every run, and inspect the dataset folder before any HuggingFace upload.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/validate.py:464
Finding

PII Validator Discloses Sensitive Values in Plaintext Logs

Content
View full analysis
Dict[str, List[Tuple[int, str, str]]]: """Scan all string fields for PII patterns. Returns findings dict.""" patterns = dict(DEFAULT_PII_PATTERNS) if include_contextual: patterns.update(CONTEXTUAL_PII_PATTERNS) findings: Dict[str, List[Tuple[int, str, str]]] = { k: [] for k in patterns } for i, row in enumerate(rows): for field, val in row.items(): if not isinstance(val, str): continue for pii_type, pattern in patterns.items(): for m in pattern.finditer(val): if pii_type == "credit_card" and not _luhn_ok(m.group()): continue findings[pii_type].append((i, field, m.group())) break return findings ``` The collected sensitive value is subsequently printed without masking: ```python for row_idx, field, match in hits[:3]: print(f" row {row_idx}.{field}: {match!r}") ``` ### Technical Analysis The PII scanner collects the complete regex match through `m.group()`. This may contain an email address, telephone number, Social Security number, or Luhn-valid payment-card number. The reporting logic then writes up to three complete values from each finding category to standard output. A validation report does not need the complete sensitive value to identify the affected record. The row index, field name, PII type, and a masked suffix are sufficient for remediation. Printing the complete value violates data minimization and expands the number of systems holding the sensitive information. Although the script does not di ...[truncated 1872 chars]
Remediation
View remediation
str: if pii_type in {"us_ssn", "credit_card", "us_phone"}: digits = re.sub(r"\D", "", value) return f"***{digits[-4:]}" if len(digits) >= 4 else "***" if pii_type == "email": local, _, domain = value.partition("@") return f"{local[:1]}***@{domain}" if domain else "***" return "***" for row_idx, field, match in hits[:3]: print( f" row {row_idx}.{field}: " f"{mask_match(pii_type, match)!r}" ) ``` ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/produce_dataset.py:563
Finding

Dataset Rebuilds Can Retain and Publish Stale Parquet Shards

Content
View full analysis
Tuple[List[Path], int]: """Write rows as sharded parquet for one split.""" shards = shard_rows(rows, shard_size_bytes) n_shards = len(shards) paths: List[Path] = [] total_nbytes = 0 for i, shard_rows_list in enumerate(shards): ds = _rows_to_dataset(shard_rows_list, features) fname = f"{split_name}-{i:05d}-of-{n_shards:05d}.parquet" fpath = out_dir / fname if quiet: import contextlib import io with contextlib.redirect_stderr(io.StringIO()), \ contextlib.redirect_stdout(io.StringIO()): ds.to_parquet(str(fpath)) else: ds.to_parquet(str(fpath)) paths.append(fpath) return paths, total_nbytes ``` There is no cleanup, manifest comparison, non-empty-directory rejection, or atomic replacement before current shards are written. ### Technical Analysis Shard names include the current total shard count. If an earlier build generated more shards, a later smaller build writes new filenames but does not remove the old files. The same problem occurs when split names change or a split is removed. For example: ```text First build: train-00000-of-0000 ...[truncated 2443 chars]
Remediation
View remediation
None: out_dir.mkdir(parents=True, exist_ok=True) existing_shards = list(out_dir.glob("*.parquet")) if existing_shards and not overwrite: raise FileExistsError( "Output directory contains existing Parquet shards; " "use --overwrite or choose a fresh directory." ) if overwrite: for shard in existing_shards: shard.unlink() ``` For stronger safety, write the full build to a temporary directory and replace the destination only after validation succeeds. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (15)

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding

The code substantially matches part of the declared purpose: it does build HuggingFace-compatible parquet shards, handles schema/features parsing, split strategies, deduplication, sharding, and statistics. However, the declared description claims a complete, publish-ready dataset pipeline including README.md with YAML frontmatter, dataset card generation, and provenance script creation. None of those capabilities are present in this code chunk. The script writes parquet data and stats.json only. Therefore the description overstates the implemented functionality in a material way, making this a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The description presents a full dataset production pipeline, but the supplied code chunk only validates raw input data before sharding. Its own docstring explicitly says it is a 'Standalone validation suite' to use before running produce_dataset.py. While validation is one sub-capability mentioned in the description, this code does not implement the core claimed behavior of assembling and publishing a complete HuggingFace-format dataset. Therefore the code’s primary purpose is materially narrower and different from the declared purpose.

Content

No source excerpt is available for this finding.

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
80% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · scripts/validate.py (reported line 559)May include surrounding context.

python
for w in warnings:
            print(f"    - {w}")
    else:
        print("  ✓ No warnings")
    if critical_failure:
        print("\n  ✗ Critical failure — fix before proceeding to produce_dataset.py")
        sys.exit(1)

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
91% confidence
Finding

The skill explicitly instructs the agent to read local files, write dataset artifacts, and run shell commands, but it declares no tool scope or permissions boundary. In an agent framework, that creates an authorization gap where a broadly-triggered skill can access or modify the filesystem or execute commands without an explicit least-privilege contract.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The trigger phrases are extremely broad (create, build, package, publish, make a dataset) and can cause the skill to activate in contexts where the user did not intend a filesystem-writing, shell-capable workflow. When combined with undeclared tool scope, over-triggering increases the chance of unintended data access, file creation, or execution of processing pipelines on sensitive inputs.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

This skill is designed to assemble and potentially publish datasets, yet the top-level instructions do not prominently warn against processing or publishing sensitive, regulated, or personal data without explicit review and consent. In this context, omission matters because the workflow includes collection, packaging, and optional hub publication, which can turn accidental inclusion of PII or confidential data into an irreversible disclosure.

Content

No source excerpt is available for this finding.

Scope Creep

Low
Category
Excessive Agency
Confidence
70% confidence
Finding

Skill's behavior or capabilities extend beyond its stated purpose. Scope creep allows an agent to perform actions unrelated to its documented functionality, increasing the attack surface.

Content

Scanner excerpt · assets/licenses/mit.txt (reported line 16)May include surrounding context.

text
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

Line L051 instructs that the language: field must be a list of ISO 639-1 codes, which imposes a specific language/locale representation as a blanket requirement. This is a natural-language locale policy constraint with no opt-in, exception handling, or justification for datasets that may require other locale conventions.

Content

No source excerpt is available for this finding.

Unverifiable Dependency: datasets has 1 known advisory(ies) (CVE-2026-66007 (Datasets through 5.0.0, fixed in commit f989ef9, contains a path traversal vulne)), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding

Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
50% confidence
Finding

Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Content

Scanner excerpt · requirements.txt (reported line 18)May include surrounding context.

text
# PyArrow — parquet backend used by datasets. Pinned to a recent version
# because the YAML `dataset_size` computation depends on
# `Dataset.data.table.nbytes` which behaves consistently across 14+.
pyarrow>=14.0

# PyYAML — used by make_card.py to parse and validate the YAML frontmatter
# in the generated README.md.

Unverifiable Dependency: pyarrow has 9 known advisory(ies) (CVE-2023-47248 (PyArrow: Arbitrary code execution when loading a malicious data file); CVE-2019-12408 (Missing Initialization of Resource in Apache Arrow); CVE-2019-12410 (Missing Initialization of Resource in Apache Arrow) +6 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding

Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
50% confidence
Finding

Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Content

Scanner excerpt · requirements.txt (reported line 22)May include surrounding context.

text
# PyYAML — used by make_card.py to parse and validate the YAML frontmatter
# in the generated README.md.
pyyaml>=6.0

# Optional (not installed by default):
#

Unverifiable Dependency: pyyaml has 8 known advisory(ies) (CVE-2019-20477 (Deserialization of Untrusted Data in PyYAML); CVE-2020-1747 (Improper Input Validation in PyYAML); CVE-2020-14343 (Improper Input Validation in PyYAML) +5 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
40% confidence
Finding

Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
92% confidence
Finding

The file sets LANGUAGES = ["en"], which is a natural-language locale constraint embedded in code. The policy says to flag language or locale restrictions unless the skill offers user opt-in or clearly documents a justified region- or language-specific limitation, which this file does not.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
87% confidence
Finding

The script calls datasets.load_dataset() on a user-supplied local path without constraining behavior. In the Hugging Face ecosystem, dataset loading can invoke builder logic, parse repository metadata, and potentially perform network access or execute repository-provided loading code depending on dataset structure and library settings. Because this is a smoke-test utility intended to be run on arbitrary produced datasets, that makes the operation riskier than a purely local file validation step.

Content

No source excerpt is available for this finding.