T09 · Insecure Skill Coding Practices
- Location
scripts/ingest.py:39- Finding
Detected Sensitive Data Is Persisted and Exported in Plaintext
- Content
View full analysis
Vulnerability Details
File Location:
scripts/ingest.py:39-45,scripts/ingest.py:91-96,scripts/ingest.py:221-223, andscripts/export_training.py:119-128
Vulnerability Type: Plaintext storage and export of sensitive information
Risk Level: MediumVulnerable Code
python # scripts/ingest.py:39-45 PII_PATTERNS = [ (re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), 'SSN'), (re.compile(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b'), 'credit_card'), (re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b'), 'email'), (re.compile(r'\b(?:password|passwd|pwd)\s*[:=]\s*\S+', re.IGNORECASE), 'password'), (re.compile(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b'), 'phone'), ]python # scripts/ingest.py:91-96 pii_flags = scan_pii(messages) if pii_flags: print(f' ⚠️ PII detected: {", ".join(sorted(pii_flags))}') else: print(f' PII: none detected')python # scripts/ingest.py:221-223 with open(sources_dir / filename, 'w', encoding='utf-8') as f: for msg in messages: f.write(json.dumps(msg, ensure_ascii=False) + '\n')python # scripts/export_training.py:119-128 for src_file in sources_dir.iterdir(): if src_file.name.startswith('.'): continue if src_file.suffix not in ('.jsonl', '.txt', '.json', '.csv'): continue dst = raw_dir / src_file.name shutil.copy2(src_file, dst) stats['files'] += 1Technical Analysis
The ingestion pipeline explicitly recognizes Social Security numbers, payment-card numbers, email addresses, passwords, and telephone numbers. Detection, however, only produces a console warning. It does not redact, quarantine, encrypt, exclude, or require confirmation before retaining the matching content.
The complete message is subsequently written to
sources/*.jsonl, submitted to MemPalace storage, and potentially copied into the export directory undertraining/raw/. The files and directories are created without explicit owner-only permi ...[truncated 1890 chars]- Remediation
View remediation
Remediation Suggestions
-
Introduce a configurable PII policy with secure defaults:
reject: stop ingestion when high-risk data is found.redact: replace matched values before persistence.quarantine: retain affected entries separately with restricted access.allow: require explicit, informed user confirmation.
-
Treat passwords, SSNs, and payment-card matches as high severity and exclude them from both MemPalace storage and training exports by default.
-
Apply redaction before writing source backups or passing messages to downstream storage. Preserve only non-sensitive metadata indicating the type and number of redactions.
-
Create knowledge and export directories with mode
0700and sensitive files with mode0600. Verify permissions after creation instead of relying on the environment's umask. -
Add an export-time PII scan so previously imported or manually modified files cannot bypass ingestion-time controls.
-
Add an option to omit
training/raw/entirely or export only a sanitized version. The destination should not retain stale sensitive files from earlier exports. -
Document data-retention, secure-deletion, backup, and export-sharing risks. Consider encryption at rest where plaintext source preservation is required.
-
Add automated tests verifying that each supported PII class is rejected or redacted in source backups, MemPalace input, conversations, and raw exports.
-