Security audit

Vector Store Shootout

Security checks for vulnerabilities and agentic risk

Overview

This appears to be a real vector-store comparison skill, but it under-discloses network text sharing and has unsafe PostgreSQL table handling that could expose or delete data.

Review before installing. Use only non-sensitive test data unless you deliberately configure where embeddings and vector databases run, avoid remote Ollama/OpenAI for private corpora, do not pass untrusted pgvector table names, and change the documented database credentials and anonymous-access settings before any shared or networked deployment.

A.I.G

Vulnerability Patterns

Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (3)

T09 · Insecure Skill Coding Practices

Warning

Location: scripts/numpy_store.py:164
Finding: Undeclared transmission of document and query contents to configurable network services
Content: View full analysis
list[list[float]]: """Priority: embed_fn (tests) → Ollama → OpenAI → TF-IDF.""" if self._embed_fn: return self._embed_fn(texts) if self._ollama_url: try: return self._ollama_embed(texts) except Exception as exc: logger.warning("Ollama embed failed, trying OpenAI: %s", exc) if self._openai_key: try: return self._openai_embed(texts) except Exception as exc: logger.warning("OpenAI embed failed, using TF-IDF: %s", exc) logger.warning("All embedding provide ...[truncated 3576 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: scripts/pgvector_store.py:70
Finding: SQL injection and unintended table deletion through an unsafe PostgreSQL table identifier
Content: View full analysis
%s::vector) AS similarity FROM {self._table_name} ORDER BY embedding <=> %s::vector LIMIT %s """, (vec_str, vec_str, k), ) ``` ```python def cleanup(self) -> None: """Drop the isolated table and close the connection.""" if self._conn and self._created: try: with self._conn.cursor() as cur: cur.execute(f"DROP TABLE IF EXISTS {self._table_name}") self._conn.commit() logger.debug("pgvector: dropped table %s", self._table_name) ``` ```python with conn.cursor() as cur: cur.execute( f""" CREATE TABLE IF NOT EXISTS {self._table_name} ( id SERIAL PRIMARY KEY, doc_id TEXT NOT NULL, text TEXT NOT NULL, embedding vector({_VECTOR_SIZE}) ) """ ) ``` ### Technical Analysis SQL value placeholders cannot safely represent SQL identi ...[truncated 2584 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Warning

Location: scripts/pgvector_store.py:13
Finding: Documented database deployment defaults expose data through weak credentials and anonymous access
Content: View full analysis
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands

Findings (48)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: This is a strong description-behavior mismatch: the skill claims to provide eight interchangeable vector-store backends, but the analysis indicates only a numpy backend is implemented and that outbound calls to Ollama localhost and the OpenAI embeddings API occur without clear disclosure. Hidden network behavior and overstated capabilities are dangerous because users may expose data to external services or make trust decisions based on false assumptions about locality, privacy, and backend parity.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The docstring promises 'No external API calls in the indexing path,' yet the actual indexing flow invokes _embed(), which can call remote services with raw texts. In security-sensitive or regulated environments, this kind of deceptive or inaccurate disclosure can directly cause confidential data exfiltration and compliance violations.
Content: No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

Category: MCP Least Privilege
Confidence: 84% confidence
Finding: The skill metadata does not declare an explicit tool scope or permissions model, yet the package is described as including backends that commonly require client-server connectivity such as Qdrant, Weaviate, and Milvus. Even though the metadata says outbound is false and frames network use as deployment-dependent, the absence of clear permission declarations can mislead users about what the skill may access when those backends are used.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 93% confidence
Finding: This store implementation transmits document/query text to embedding providers, which goes beyond a narrowly described 'vector store backend' and creates a data-flow risk not obvious from the skill description. If users expect only local storage benchmarking, sensitive corpus contents may be disclosed to local or third-party services during indexing and querying.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 82% confidence
Finding: This POST request sends texts to the configured Ollama endpoint. Although the default target is localhost and therefore less severe than a public SaaS endpoint, the URL is configurable and could point to a remote host, causing unexpected disclosure of indexed/query content.
Content: Scanner excerpt · scripts/lancedb_store.py (reported line 182)May include surrounding context.
python
def _ollama_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( self._ollama_url, json={"model": _OLLAMA_EMBED_MODEL, "input": texts}, timeout=60,

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: User-supplied texts are sent directly to embedding APIs at both Ollama and OpenAI call sites, but the code provides no explicit warning, consent prompt, or classification check before transmitting content. This is dangerous because RAG inputs often contain sensitive internal data, and users may not realize indexing/query text leaves the local process or machine.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The OpenAI fallback sends input texts off-host to a remote SaaS endpoint, which is materially broader than comparing interchangeable vector-store backends. In environments using this skill for internal RAG evaluation, proprietary documents and queries could be exfiltrated to an external provider without clear consent or policy checks.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 97% confidence
Finding: The JSON payload to the OpenAI embeddings endpoint includes raw input texts, which creates a direct external data transmission path. In the context of a benchmarking skill, this hidden dependency on remote processing increases privacy and compliance risk beyond the apparent storage-only purpose.
Content: Scanner excerpt · scripts/lancedb_store.py (reported line 192)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 97% confidence
Finding: The JSON payload to the OpenAI embeddings endpoint includes raw input texts, which creates a direct external data transmission path. In the context of a benchmarking skill, this hidden dependency on remote processing increases privacy and compliance risk beyond the apparent storage-only purpose.
Content: Scanner excerpt · scripts/lancedb_store.py (reported line 192)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 96% confidence
Finding: The hardcoded OpenAI endpoint indicates intentional support for external transmission to a third-party cloud service. That is risky in this skill context because users evaluating vector stores may unknowingly send sensitive corpora or queries outside their environment.
Content: Scanner excerpt · scripts/lancedb_store.py (reported line 193)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"}, timeout=30,

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The file-level documentation states there are no external API calls in the indexing path, but add_documents() calls _embed(), which may send full document text to Ollama or OpenAI. This mismatch creates a real privacy and security risk because operators may index sensitive data under a false assumption that processing is entirely local.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 80% confidence
Finding: This request transmits text to the configured Ollama endpoint. While often intended for a local service, the code accepts an arbitrary URL and uses plain HTTP by default, so the transmission can leave the local machine or traverse an untrusted network if misconfigured.
Content: Scanner excerpt · scripts/lightrag_store.py (reported line 240)May include surrounding context.
python
def _ollama_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( self._ollama_url, json={"model": _OLLAMA_EMBED_MODEL, "input": texts}, timeout=60,

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The OpenAI embedding path sends the supplied texts to a third-party external API, which may include proprietary or sensitive corpus content. Without any user-facing disclosure, consent mechanism, or data-classification guardrail, this can result in unintended data exposure outside the trust boundary.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 92% confidence
Finding: This is the same external transmission path to OpenAI's embeddings service, carrying user-provided texts beyond the local environment. Because the skill presents itself as a vector-store component and the docstring understates network behavior, the risk of accidental sensitive-data disclosure is materially increased.
Content: Scanner excerpt · scripts/lightrag_store.py (reported line 250)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 92% confidence
Finding: This is the same external transmission path to OpenAI's embeddings service, carrying user-provided texts beyond the local environment. Because the skill presents itself as a vector-store component and the docstring understates network behavior, the risk of accidental sensitive-data disclosure is materially increased.
Content: Scanner excerpt · scripts/lightrag_store.py (reported line 250)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 91% confidence
Finding: The hard-coded OpenAI API endpoint confirms that document/query text may be sent to an external third-party service. In combination with the misleading file description and lack of user-facing warning, this can cause unintentional disclosure of proprietary or regulated data.
Content: Scanner excerpt · scripts/lightrag_store.py (reported line 251)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"}, timeout=30,

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 92% confidence
Finding: The skill metadata frames this component as a vector-store backend, but the implementation also performs embedding generation and can send document text to external services. That mismatch is security-relevant because users may supply sensitive content expecting only local storage/comparison behavior, while the code introduces undisclosed network data flows.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The OpenAI fallback adds third-party network exfiltration of input texts that is not reflected in the stated purpose of comparing vector stores. Because the fallback is automatic when local embedding fails, users could unintentionally transmit sensitive documents to an external API without realizing this backend does more than storage/search.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 80% confidence
Finding: This call posts texts to a configurable Ollama endpoint, which is an external transmission from the process even if the default points to localhost. The risk is lower than a third-party SaaS call because the default target is local, but the URL is configurable and could point to a remote host, making data disclosure possible without additional safeguards.
Content: Scanner excerpt · scripts/milvus_store.py (reported line 251)May include surrounding context.
python
def _ollama_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( self._ollama_url, json={"model": _OLLAMA_EMBED_MODEL, "input": texts}, timeout=60,

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: The code transmits document text to embedding services at the call sites with no explicit user-facing warning or consent mechanism. Since embeddings are generated from raw input text, this can leak proprietary, regulated, or secret data to local or external services unexpectedly, especially through fallback behavior.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: A remote third-party embedding path is not necessary for the stated goal of evaluating vector stores, so it expands the trust boundary without clear justification. In this context, sending raw document text to OpenAI increases privacy and compliance risk while not being inherent to the vector-store comparison function.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 98% confidence
Finding: The code includes an authenticated POST to OpenAI containing the input texts, which is a concrete outbound data transfer to a third party. In a skill advertised as a vector-store backend, this undisclosed behavior makes accidental data leakage more likely because callers may not anticipate remote processing.
Content: Scanner excerpt · scripts/milvus_store.py (reported line 261)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 98% confidence
Finding: The code includes an authenticated POST to OpenAI containing the input texts, which is a concrete outbound data transfer to a third party. In a skill advertised as a vector-store backend, this undisclosed behavior makes accidental data leakage more likely because callers may not anticipate remote processing.
Content: Scanner excerpt · scripts/milvus_store.py (reported line 261)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"},

External Transmission

Medium

Category: Data Exfiltration
Confidence: 97% confidence
Finding: The hardcoded OpenAI API endpoint evidences a built-in third-party data egress path. In this skill context, that is more dangerous because the advertised function is backend comparison, not cloud-based text processing, so users may unknowingly expose sensitive corpus contents.
Content: Scanner excerpt · scripts/milvus_store.py (reported line 262)May include surrounding context.
python
def _openai_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( "https://api.openai.com/v1/embeddings", headers={"Authorization": f"Bearer {self._openai_key}"}, json={"input": texts, "model": "text-embedding-3-small"}, timeout=30,

External Transmission

Medium

Category: Data Exfiltration
Confidence: 80% confidence
Finding: The Ollama embedding path transmits input texts over HTTP to a service endpoint, which is an external transmission of potentially sensitive data. In this specific context the default target is localhost, so exposure is materially lower than an Internet service, but use of a configurable URL and plaintext HTTP can still leak data if pointed at a remote host or if local traffic is not trusted.
Content: Scanner excerpt · scripts/numpy_store.py (reported line 185)May include surrounding context.
python
def _ollama_embed(self, texts: list[str]) -> list[list[float]]: import requests resp = requests.post( self._ollama_url, json={"model": _OLLAMA_EMBED_MODEL, "input": texts}, timeout=60,

Static analysis

No suspicious patterns detected.