Security audit

Sharedintellect Quorum

Security checks for vulnerabilities and agentic risk

Overview

This appears to be a real AI review tool, but it needs review because it can send full file contents to external model providers, installs mutable code and dependencies, and uses persistent local memory that can influence future reviews.

Before installing, treat Quorum as a tool that may transmit the files you review to whichever LLM provider you configure. Do not run it on secrets, customer data, or proprietary code unless that provider and policy are acceptable. Prefer installing in an isolated environment from a reviewed commit with pinned dependencies, and consider using --no-learning or removing project-local known_issues.json when auditing untrusted repositories.

A.I.G

Vulnerability Patterns

Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Insecure DependenciesIntroduces malicious components through unsafe dependency sources
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review

Findings (3)

T02 · Agent Memory Poisoning

Error

Location: reference-implementation/quorum/learning.py:34
Finding: Project-local learning memory enables persistent prompt injection
Content: View full analysis
list[Issue]: """Load issues from JSON file. Returns empty list if file does not exist.""" if not self._path.exists(): return [] try: raw = self._path.read_text(encoding="utf-8") data = json.loads(raw) return [Issue(**item) for item in data] except Exception as e: logger.warning("Failed to load %s: %s", self._path, e) return [] ``` `reference-implementation/quorum/learning.py:161-175` ```python def to_critic_context(self) -> str: """ Format mandatory issues as brief instructions for critic system prompts. Returns empty string if no mandatory issues exist. """ mandatory = self.get_mandatory() if not mandatory: return "" lines = ["## Known Recurring I ...[truncated 2805 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: reference-implementation/quorum/pipeline.py:423
Finding: Sensitive artifacts are transmitted unredacted to external LLM providers
Content: View full analysis
str: """Call any LiteLLM-supported model and return text.""" logger.debug("Calling model=%s (temp=%.2f, max_tokens=%d)", model, temperature, max_tokens) try: response = completion( model=model, messages=messages, temperature=temperature, max_tokens=max_tokens, **self._extra_kwargs, ) ``` ### Technical Analysis The pre-screen performs deterministic checks for credentials and PII, but the reviewed control flow continues into the supervisor and critics regardless ...[truncated 2108 chars]
Remediation: View remediation

T08 · Insecure Dependencies

Warning

Location: SKILL.md:4
Finding: Installation relies on mutable repository state and unpinned dependencies
Content: View full analysis
=1.40.0 pydantic>=2.0.0 pyyaml>=6.0.0 click>=8.0.0 ``` ### Technical Analysis The installation command clones the repository's mutable default branch rather than a reviewed commit or signed release. It then installs dependencies using lower-bound-only version constraints without a lock file or package hashes. This means two installations performed at different times may execute different code despite using the same Skill metadata. A compromise of the upstream repository, an unsafe future package release, or a dependency-resolution change could introduce attacker-controlled installation or runtime code after the Skill package was reviewed. The audit did not identify a currently malicious dependency, dependency-confusion package name, or confirmed upstream compromise. The finding concerns the absence of controls needed to make installation reproducible and resistant to future supply-chain changes. ### Attack Path 1. An attacker compromises the upstream repository, its release process, ...[truncated 1045 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (149)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description suggests an operational multi-agent AI review system. The actual code chunk does not implement or invoke multiple AI critics, rubric-based critique generation, or evidence-grounded artifact assessment. Instead, it is exclusively a unit/integration test file for a scoring framework: it constructs temporary artifact/annotation files, validates SHA-256 hashes, parses YAML annotations, tests finding-matching heuristics, trap handling, and aggregate metric computation. This is a materially different primary purpose. The filesystem interactions are consistent with testing infrastructure, but they are still absent from the declared permissions. Overall, this is a strong description-behavior mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The description presents an AI-based multi-agent validation system with six independent critics evaluating artifacts against rubrics. The code instead is a single-process prescreen tool that reads one local file and applies deterministic checks such as regex scans for secrets/PII, JSON/YAML/Python parsing, markdown link existence, TODO markers, whitespace, and empty-file detection. It produces a JSON summary, but there is no evidence of agents, model calls, rubric evaluation, quorum logic, or independent critiques. This is a materially different primary purpose and capability set, so the description does not accurately represent the code.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The description claims an AI-based multi-agent validation system with six independent critics evaluating artifacts against rubrics. The supplied code does not implement agents, model calls, rubric-based reasoning, or evidence-grounded critique in that sense. Instead, it performs local regex/parsing-based prescreen checks on files and reports structured PASS/FAIL/SKIP results. Its primary purpose is materially different: deterministic static prescreening/linting rather than multi-agent AI validation. The code also includes a repository-wide markdown link scan mode, an undeclared operational capability.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description describes an AI-based multi-agent validation system with six independent critics evaluating artifacts. The actual code does not implement or invoke such behavior. Instead, it is a unit test file for a filesystem/documentation link scanner that checks Markdown links across files in a repo-like directory structure. Its primary purpose, resources accessed, and functionality are materially different from the declared purpose. Therefore this is a clear mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description emphasizes a validation framework with six independent AI critics evaluating artifacts against rubrics and producing evidence-grounded findings. This code chunk does not implement critic evaluation or rubric-based validation. Instead, it implements a downstream remediation component: a Fixer agent that consumes existing findings and proposes concrete text replacements for severe issues. That is a materially different primary purpose and adds an undeclared capability (automated fix proposal generation). No problematic extra permissions or triggers are evident, but the behavior of this module is not accurately represented by the declared description.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 92% confidence
Finding: The code is clearly related to the declared domain: it is part of a validation framework that evaluates artifacts against rubrics using critic agents. However, the description claims '6 independent AI critics,' while this code chunk only registers four critics (correctness, completeness, security, code_hygiene), with others explicitly noted as future additions or handled elsewhere. The primary function of this file is supervisor/orchestration—domain classification, critic instantiation, prompt/context injection, and threaded execution—not the full six-critic evaluation system as described. There are no suspicious undeclared external capabilities or resource accesses; the mismatch is mainly that the declared description overstates what this supplied code chunk actually implements.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The declared description says the skill is a multi-agent validation framework whose primary function is to have six independent AI critics evaluate artifacts against rubrics. The supplied code does not implement critic orchestration, rubric-based evaluation, artifact analysis, or evidence-grounded finding generation. Instead, it is a reporting utility focused on post-run audit CSV generation: it reads artifact.txt, run-manifest.json, and verdict.json from run directories, computes hashes, file sizes, token estimates, durations, token/cost aggregates, model lists, and pass/fail counts, then writes summary/detail CSVs. This is a materially different primary purpose from the declared one, so it is a clear description-behavior mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 91% confidence
Finding: The overall domain is related: the CLI clearly supports artifact validation against rubrics with evidence-oriented output, so the description is directionally aligned. However, the specific declared description is not accurate for this code chunk. First, the code does not show a '6 independent AI critics' framework; instead it loads critics from configuration and the first-run default config sets only ['correctness', 'completeness'], i.e. two critics, with no evidence here of six independent agents. Second, the code exposes several significant capabilities not reflected in the description: configuration initialization, API-key setup, rubric management, issue/learning-memory management, batch resume, cost/time estimation, budget caps, and audit/fix-loop support. These are substantial product behaviors beyond a simple multi-agent validation framework description. Therefore this chunk materially differs from the declared purpose enough to count as a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 92% confidence
Finding: The declared description emphasizes the core validation behavior: six AI critics evaluating artifacts against rubrics. The supplied code chunk does not implement critic orchestration, rubric evaluation, evidence-grounded findings, or specifically six agents. Instead, it provides ancillary infrastructure for cost tracking, budget checking, and runtime estimation for such validation runs. These are related to the broader system, but they are materially different capabilities from the declared purpose and are not mentioned in the description. Therefore this chunk's behavior does not accurately match the declared description.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 90% confidence
Finding: The description claims a broader 'multi-agent validation framework' with 6 independent AI critics. This code chunk is not that framework; it is one concrete critic class, CrossConsistencyCritic, for Phase 2 cross-file consistency analysis. It evaluates file relationships like implements, documents, delegates, and schema_contract, builds prompts for an LLM, handles missing files, and parses structured findings. While this is related to validation and evidence-grounded findings, the primary purpose and scope are materially narrower and different from the declared description. No undeclared dangerous permissions/resources are evident, but the stated functionality does not accurately represent this specific code chunk.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 93% confidence
Finding: The code does not directly implement the core declared behavior of '6 independent AI critics evaluate artifacts against rubrics with evidence-grounded findings.' Instead, it implements a stateful persistence and learning subsystem that stores issue history in JSON, updates frequencies from findings, promotes patterns based on thresholds, and injects mandatory recurring checks into future critic prompts. This is a meaningful undeclared capability because it adds persistent cross-run memory and prompt-shaping behavior, neither of which is represented in the declared description or permissions. While this may support a broader validation framework, the code chunk’s primary purpose is materially different from the declared purpose for this skill description.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 87% confidence
Finding: The code broadly fits a validation framework that evaluates artifacts against rubrics and produces findings, so the core theme is aligned. However, the declared description is materially narrower and more specific than the code shown. The implementation is not just a simple '6 independent AI critics' validator: it includes a supervisor/aggregator pipeline, optional prescreening, automatic fix proposal application and revalidation loops, cross-artifact consistency checks, tester verification, persistent learning-memory updates, cost tracking, audit CSV generation, and robust batch/resume orchestration. Also, this file does not substantiate the exact claim of '6 independent AI critics'; critics are selected through a supervisor and registry, and the number/independence is not enforced here. These are substantive undeclared capabilities and a partially mismatched characterization of how the system operates.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The declared description emphasizes a 6-agent AI critic framework that evaluates artifacts against rubrics. The supplied code chunk does not implement multi-agent orchestration, critic coordination, rubric evaluation, or anything specific to '6 independent AI critics.' Instead, it is a pre-screen component that runs deterministic, mostly regex/parser-based checks and optional external static-analysis tools before LLM dispatch. While this may support the broader framework, the actual code’s primary purpose in this chunk is materially different from the declared description and includes undeclared capabilities such as subprocess-based execution of security tools and direct static scanning for credentials, PII, syntax, and broken links.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 93% confidence
Finding: The declared description presents a specific end-user capability: a six-agent validation framework that evaluates artifacts against rubrics. The supplied code chunk instead contains only a low-level abstract provider interface for making LLM calls and returning text or JSON. While such a provider could support the larger framework, this chunk does not itself perform or express the declared validation behavior. Therefore the code's actual purpose is materially different from the declared purpose for this skill.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description emphasizes a specific application-level behavior: a six-agent validation framework with independent critics evaluating artifacts against rubrics. The supplied code does not implement multi-agent validation, critic orchestration, rubric evaluation, or evidence-grounded findings. Instead, it is an infrastructure component: a generic LiteLLM provider wrapper for calling many third-party models, handling auth via environment variables, parsing JSON outputs, and tracking usage/cost. While such a provider could support a larger multi-agent framework, this chunk’s actual purpose is materially different and includes undeclared capabilities related to outbound model access and credential handling.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description describes a full multi-agent evaluation framework with six AI critics producing findings. The supplied code chunk does not implement agent orchestration, artifact evaluation, rubric scoring, or evidence-grounded critique generation. Instead, it is a narrow utility for loading rubric definitions from JSON files on disk and converting them into Rubric/RubricCriterion objects. This is a materially different primary purpose from the declared behavior. Filesystem access to rubric files is present in the code but not reflected in the declared permissions, though the main mismatch is functional scope and purpose.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 90% confidence
Finding: The code chunk is a narrow helper tool for structural validation of JSON/YAML content. It loads files from disk, parses them, checks required keys and types, optionally uses jsonschema, and formats validation errors. It does not implement or demonstrate a multi-agent framework, six critics, rubric-based artifact evaluation, or evidence-grounded critique generation beyond simple schema violations. While such a tool could support a validation framework, this chunk’s actual behavior is materially different from the declared primary purpose, so this is a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description says this skill is a multi-agent AI validation framework with six independent critics evaluating artifacts against rubrics. The supplied code does nothing of that sort. It is a generic god-function test fixture that validates input fields, transforms values, writes results to disk, and optionally executes arbitrary code from input using eval. Its primary purpose and behavior are materially different from the declared purpose, and it includes undeclared file access and dangerous code execution capabilities. This is a clear mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description describes an AI-based multi-agent validation framework, but the provided code is only a filesystem-oriented test fixture for hardcoded path detection. It defines constant paths and reads a local file from a hardcoded directory. There is no implementation of multiple agents, critique/evaluation logic, rubric processing, or evidence-grounded findings. This is a materially different primary purpose and includes undeclared filesystem access.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description describes an AI-driven multi-agent validation system with six critics and rubric-based artifact assessment. The supplied code does not implement any such framework, agents, evaluation logic, rubric handling, or evidence collection. Instead, it is a small utility module for basic data representation and list processing. This is a materially different primary purpose, so it should be flagged as a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description describes a sophisticated multi-agent validation system involving six independent AI critics and rubric-based evaluation. The supplied code does none of this. It only extracts fields from an input dictionary and returns a doubled numeric value. This is a materially different primary purpose, so it is a clear mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description emphasizes the core product behavior: six independent AI critics evaluating artifacts against rubrics. The supplied code chunk does not implement or primarily exercise that multi-agent evaluation behavior. Instead, it focuses on audit/reporting infrastructure: building CSV detail and summary rows, computing durations, counting findings, aggregating model costs and tokens, generating single-file and batch audit reports, exposing an audit-report CLI flag, and verifying that pipeline runs write audit files and artifact hashes. While this may be adjacent supporting functionality for the broader system, the actual code chunk’s primary purpose is audit report generation/testing rather than multi-agent validation by 6 critics. No hidden dangerous permissions are evident, but the behavior is materially different from the declared purpose.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 94% confidence
Finding: The declared description emphasizes a multi-agent validation framework with 6 independent AI critics evaluating artifacts against rubrics. The supplied code chunk does not implement that primary behavior; instead, it is specifically a test module focused on operational resilience of batch execution in the validation pipeline. Its concerns are persistence, manifest integrity, recovery from crashes, and interruption handling. While it references validation-related models and mocks supervisor/aggregator/provider components, the substantive behavior in this chunk is unrelated to the claimed six-critic evaluation functionality. Therefore the description does not accurately represent what this code chunk actually does.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description is about the product's high-level function: a multi-agent validation framework with 6 AI critics evaluating artifacts. The supplied code chunk does not implement that behavior. Instead, it contains unit tests for a command-line interface, covering help text, version output, run command validation, rubrics subcommands, config initialization, and error handling. While CLI tests may support such a framework, this specific chunk's actual purpose is materially different from the declared purpose, so it should be flagged as a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: The declared description emphasizes a runtime multi-agent evaluation system with six independent AI critics assessing artifacts. The supplied code does not implement or invoke any multi-agent criticism, rubric evaluation, evidence generation, or artifact analysis. Instead, it is strictly unit-test code for configuration parsing and validation. While configuration support could be part of a larger multi-agent framework, this specific chunk’s primary behavior is materially different from the declared purpose, so it should be flagged as a mismatch.
Content: No source excerpt is available for this finding.

Static analysis

Detected: suspicious.dynamic_code_execution, suspicious.exposed_secret_literal, suspicious.obfuscated_code (+1 more)

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: golden-test-set/tests/test_score.py:197

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: reference-implementation/quorum/critics/code_hygiene.py:136

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: reference-implementation/quorum/critics/security.py:216

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: reference-implementation/tests/fixtures/bad/code-god-function.py:34

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: reference-implementation/tests/test_models.py:389

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: reference-implementation/examples/sample-agent-config.yaml:71

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: reference-implementation/tests/fixtures/edge/with-api-key.txt:4

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: reference-implementation/tests/test_prescreen.py:117

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: reference-implementation/tests/test_security_accuracy.py:31

Potential obfuscated payload detected.

Warn

Code: suspicious.obfuscated_code
Location: reference-implementation/tests/test_prescreen_properties.py:106

Prompt-injection style instruction pattern detected.

Warn

Code: suspicious.prompt_injection_instructions
Location: docs/architecture/IMPLEMENTATION.md:122

Prompt-injection style instruction pattern detected.

Warn

Code: suspicious.prompt_injection_instructions
Location: ports/claude-code/quorum-validation-skill/completeness-findings.md:50