Vector Store Shootout

• Outcome: Review
• Latest audit: Sep 11, 2026 · 7:15 AM UTC
• Version: 1.0.0

Overview

This appears to be a real vector-store comparison skill, but it under-discloses network text sharing and has unsafe PostgreSQL table handling that could expose or delete data.

Review before installing. Use only non-sensitive test data unless you deliberately configure where embeddings and vector databases run, avoid remote Ollama/OpenAI for private corpora, do not pass untrusted pgvector table names, and change the documented database credentials and anonymous-access settings before any shared or networked deployment.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (3)

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/numpy_store.py:164
• Finding: Undeclared transmission of document and query contents to configurable network services<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md:1-7`; `scripts/numpy_store.py:164-202`; `scripts/lancedb_store.py:166-199`; `scripts/lightrag_store.py:223-258`; `scripts/milvus_store.py:234-269`; `scripts/pgvector_store.py:206-241`; `scripts/qdrant_store.py:210-251`; `scripts/weaviate_store.py:225-260` **Vulnerability Type**: Undeclared outbound data transmission and insufficient endpoint restrictions **Risk Level**: Medium ### Relevant Code The Skill metadata declares that outbound networking is disabled: ```yaml --- name: vector-store-shootout version: 1.0.0 description: 8 vector store implementations behind a common interface — numpy, lancedb, qdrant, pgvector, weaviate, weaviate_hybrid, milvus, lightrag. Use when evaluating RAG backends, building vector search, or comparing embedding stores. Each backend is a drop-in replacement via the base class. metadata: {"openclaw": {"emoji": "🔍", "requires": {"bins": ["python3"], "env": []}, "primaryEnv": null, "network": {"outbound": false, "reason": "All backends run locally. Network calls depend on your deployment (e.g. managed Qdrant Cloud)."}}} --- ``` However, the backends send the complete `texts` list to a configurable Ollama endpoint and, when explicitly configured with an API key, can fall back to OpenAI. The following representative implementation appears in `scripts/numpy_store.py`: ```python def _embed(self, texts: list[str]) -> list[list[float]]: """Priority: embed_fn (tests) → Ollama → OpenAI → TF-IDF.""" if self._embed_fn: return self._embed_fn(texts) if self._ollama_url: try: return self._ollama_embed(texts) except Exception as exc: logger.warning("Ollama embed failed, trying OpenAI: %s", exc) if self._openai_key: try: return self._openai_embed(texts) except Exception as exc: logger.warning("OpenAI embed failed, using TF-IDF: %s", exc) logger.warning("All embedding provide ...[truncated 3576 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Correct `SKILL.md` so the network declaration explicitly identifies: - Local Ollama HTTP access. - Optional OpenAI API access. - Connections to Qdrant, PostgreSQL, Weaviate, and Milvus services. 2. Default to an offline embedding function or require explicit network opt-in before any content is transmitted. 3. Validate `ollama_url` with a proper URL parser: - Permit only `http://localhost`, `http://127.0.0.1`, and `http://[::1]` by default. - Require a separate explicit option for remote endpoints. - Require HTTPS for non-loopback destinations. - Reject embedded credentials, unsupported schemes, redirects to disallowed hosts, and malformed URLs. 4. Disable automatic OpenAI fallback unless the caller explicitly opts into remote fallback after receiving a data-disclosure warning. 5. Provide an embedding-provider policy such as `offline`, `local`, or `remote`, with `offline` or `local` as the secure default. 6. Document that complete document and query contents—not merely derived embeddings—are sent to embedding providers. 7. Consider destination allowlisting and redirect blocking to reduce internal-network request risks. 8. Add tests confirming that offline mode makes no network calls and that non-loopback URLs are rejected unless explicitly authorized. ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/pgvector_store.py:70
• Finding: SQL injection and unintended table deletion through an unsafe PostgreSQL table identifier<![CDATA[ ## Vulnerability Details **File Location**: `scripts/pgvector_store.py:70-72, 107-110, 135-142, 156-161, 193-202` **Vulnerability Type**: SQL injection through caller-controlled identifier interpolation **Risk Level**: High ### Relevant Code The constructor accepts a caller-provided table name and only replaces hyphens: ```python raw = table_name or f"rag_{uuid.uuid4().hex[:12]}" self._table_name = raw.replace("-", "_") ``` The resulting value is directly interpolated into multiple SQL statements: ```python with conn.cursor() as cur: for text, doc_id, emb in zip(texts, auto_ids, embeddings): # Explicit ::vector cast — no register_vector adapter needed vec_str = "[" + ",".join(str(x) for x in emb) + "]" cur.execute( f"INSERT INTO {self._table_name} (doc_id, text, embedding)" f" VALUES (%s, %s, %s::vector)", (doc_id, text, vec_str), ) ``` ```python with conn.cursor() as cur: cur.execute( f""" SELECT doc_id, text, 1 - (embedding <=> %s::vector) AS similarity FROM {self._table_name} ORDER BY embedding <=> %s::vector LIMIT %s """, (vec_str, vec_str, k), ) ``` ```python def cleanup(self) -> None: """Drop the isolated table and close the connection.""" if self._conn and self._created: try: with self._conn.cursor() as cur: cur.execute(f"DROP TABLE IF EXISTS {self._table_name}") self._conn.commit() logger.debug("pgvector: dropped table %s", self._table_name) ``` ```python with conn.cursor() as cur: cur.execute( f""" CREATE TABLE IF NOT EXISTS {self._table_name} ( id SERIAL PRIMARY KEY, doc_id TEXT NOT NULL, text TEXT NOT NULL, embedding vector({_VECTOR_SIZE}) ) """ ) ``` ### Technical Analysis SQL value placeholders cannot safely represent SQL identi ...[truncated 2584 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Do not accept arbitrary table names unless required. Prefer an internally generated UUID-based identifier. 2. If caller-defined names are necessary, enforce a strict allowlist such as: - First character: ASCII letter or underscore. - Remaining characters: ASCII letters, digits, and underscores. - A conservative maximum length. 3. Compose SQL identifiers with `psycopg2.sql.Identifier` rather than f-string interpolation. For example: ```python from psycopg2 import sql cur.execute( sql.SQL("DROP TABLE IF EXISTS {}").format( sql.Identifier(self._table_name) ) ) ``` 4. Apply safe identifier composition consistently to `CREATE TABLE`, `INSERT`, `SELECT`, and `DROP TABLE`. 5. Determine whether the table existed before creation and record ownership separately, for example with `_owns_table`. Only drop the table when the current instance demonstrably created it. 6. Prefer a dedicated schema and a minimally privileged database role that can access only Skill-owned objects. 7. Provision the `vector` extension administratively instead of granting the runtime role extension-management privileges. 8. Add security tests using spaces, quotes, semicolons, comments, reserved words, schema-qualified names, and existing table names. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/pgvector_store.py:13
• Finding: Documented database deployment defaults expose data through weak credentials and anonymous access<![CDATA[ ## Vulnerability Details **File Location**: `scripts/pgvector_store.py:13-20, 43`; `scripts/weaviate_store.py:15-25` **Vulnerability Type**: Insecure service configuration and hardcoded development credentials **Risk Level**: Medium ### Relevant Code The PostgreSQL deployment example publishes the service port and assigns predictable credentials: ```yaml Docker setup (add to docker-compose.yml): pgvector: image: pgvector/pgvector:pg16 ports: ["5432:5432"] environment: POSTGRES_USER: rag POSTGRES_PASSWORD: rag POSTGRES_DB: rag volumes: - pgvector_data:/var/lib/postgresql/data ``` The same credentials are embedded in the runtime default DSN: ```python _DEFAULT_DSN = "postgresql://rag:rag@localhost:5432/rag" ``` The Weaviate deployment example publishes its ports while enabling anonymous access: ```yaml Docker setup (add to docker-compose.yml): weaviate: image: semitechnologies/weaviate:1.27.0 ports: ["8082:8080", "50051:50051"] environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true" PERSISTENCE_DATA_PATH: /var/lib/weaviate DEFAULT_VECTORIZER_MODULE: none ENABLE_MODULES: "" CLUSTER_HOSTNAME: node1 volumes: - weaviate_data:/var/lib/weaviate ``` ### Technical Analysis These configurations are presented as Docker setup instructions rather than files that the Skill deploys automatically. They are nevertheless unsafe defaults that users may copy directly. Publishing ports without an explicit loopback binding can expose services on all host interfaces, depending on Docker and firewall configuration. The PostgreSQL password is publicly predictable and reused in the default DSN. Weaviate explicitly permits anonymous access and the corresponding client code supplies no authentication credentials. The settings may be accepta ...[truncated 1561 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Replace fixed PostgreSQL credentials with generated secrets supplied through environment variables or a secrets manager. 2. Remove credentials from the default DSN. Require the caller to provide a DSN or construct it from protected configuration. 3. Bind development services to loopback explicitly: ```yaml ports: - "127.0.0.1:5432:5432" ``` ```yaml ports: - "127.0.0.1:8082:8080" - "127.0.0.1:50051:50051" ``` 4. Enable Weaviate authentication and authorization, and update the Python client to supply the required credentials. 5. Use TLS for any connection crossing a trusted local boundary. 6. Assign each service a minimally privileged account restricted to Skill-owned databases, schemas, and collections. 7. Clearly label anonymous and fixed-password examples as local-only insecure development configurations. 8. Add production deployment guidance covering firewalls, Docker network isolation, secret rotation, backups, and access logging. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (48)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 97% confidence
• Finding: This is a strong description-behavior mismatch: the skill claims to provide eight interchangeable vector-store backends, but the analysis indicates only a numpy backend is implemented and that outbound calls to Ollama localhost and the OpenAI embeddings API occur without clear disclosure. Hidden network behavior and overstated capabilities are dangerous because users may expose data to external services or make trust decisions based on false assumptions about locality, privacy, and backend parity.

Intent-Code Divergence

High

• Confidence: 98% confidence
• Finding: The docstring promises 'No external API calls in the indexing path,' yet the actual indexing flow invokes _embed(), which can call remote services with raw texts. In security-sensitive or regulated environments, this kind of deceptive or inaccurate disclosure can directly cause confidential data exfiltration and compliance violations.

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 84% confidence
• Finding: The skill metadata does not declare an explicit tool scope or permissions model, yet the package is described as including backends that commonly require client-server connectivity such as Qdrant, Weaviate, and Milvus. Even though the metadata says outbound is false and frames network use as deployment-dependent, the absence of clear permission declarations can mislead users about what the skill may access when those backends are used.

Description-Behavior Mismatch

Medium

• Confidence: 93% confidence
• Finding: This store implementation transmits document/query text to embedding providers, which goes beyond a narrowly described 'vector store backend' and creates a data-flow risk not obvious from the skill description. If users expect only local storage benchmarking, sensitive corpus contents may be disclosed to local or third-party services during indexing and querying.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,

• Confidence: 82% confidence
• Finding: This POST request sends texts to the configured Ollama endpoint. Although the default target is localhost and therefore less severe than a public SaaS endpoint, the URL is configurable and could point to a remote host, causing unexpected disclosure of indexed/query content.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: User-supplied texts are sent directly to embedding APIs at both Ollama and OpenAI call sites, but the code provides no explicit warning, consent prompt, or classification check before transmitting content. This is dangerous because RAG inputs often contain sensitive internal data, and users may not realize indexing/query text leaves the local process or machine.

Context-Inappropriate Capability

Medium

• Confidence: 95% confidence
• Finding: The OpenAI fallback sends input texts off-host to a remote SaaS endpoint, which is materially broader than comparing interchangeable vector-store backends. In environments using this skill for internal RAG evaluation, proprietary documents and queries could be exfiltrated to an external provider without clear consent or policy checks.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 97% confidence
• Finding: The JSON payload to the OpenAI embeddings endpoint includes raw input texts, which creates a direct external data transmission path. In the context of a benchmarking skill, this hidden dependency on remote processing increases privacy and compliance risk beyond the apparent storage-only purpose.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 97% confidence
• Finding: The JSON payload to the OpenAI embeddings endpoint includes raw input texts, which creates a direct external data transmission path. In the context of a benchmarking skill, this hidden dependency on remote processing increases privacy and compliance risk beyond the apparent storage-only purpose.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,

• Confidence: 96% confidence
• Finding: The hardcoded OpenAI endpoint indicates intentional support for external transmission to a third-party cloud service. That is risky in this skill context because users evaluating vector stores may unknowingly send sensitive corpora or queries outside their environment.

Description-Behavior Mismatch

Medium

• Confidence: 95% confidence
• Finding: The file-level documentation states there are no external API calls in the indexing path, but add_documents() calls _embed(), which may send full document text to Ollama or OpenAI. This mismatch creates a real privacy and security risk because operators may index sensitive data under a false assumption that processing is entirely local.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,

• Confidence: 80% confidence
• Finding: This request transmits text to the configured Ollama endpoint. While often intended for a local service, the code accepts an arbitrary URL and uses plain HTTP by default, so the transmission can leave the local machine or traverse an untrusted network if misconfigured.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The OpenAI embedding path sends the supplied texts to a third-party external API, which may include proprietary or sensitive corpus content. Without any user-facing disclosure, consent mechanism, or data-classification guardrail, this can result in unintended data exposure outside the trust boundary.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 92% confidence
• Finding: This is the same external transmission path to OpenAI's embeddings service, carrying user-provided texts beyond the local environment. Because the skill presents itself as a vector-store component and the docstring understates network behavior, the risk of accidental sensitive-data disclosure is materially increased.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 92% confidence
• Finding: This is the same external transmission path to OpenAI's embeddings service, carrying user-provided texts beyond the local environment. Because the skill presents itself as a vector-store component and the docstring understates network behavior, the risk of accidental sensitive-data disclosure is materially increased.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,

• Confidence: 91% confidence
• Finding: The hard-coded OpenAI API endpoint confirms that document/query text may be sent to an external third-party service. In combination with the misleading file description and lack of user-facing warning, this can cause unintentional disclosure of proprietary or regulated data.

Description-Behavior Mismatch

Medium

• Confidence: 92% confidence
• Finding: The skill metadata frames this component as a vector-store backend, but the implementation also performs embedding generation and can send document text to external services. That mismatch is security-relevant because users may supply sensitive content expecting only local storage/comparison behavior, while the code introduces undisclosed network data flows.

Description-Behavior Mismatch

Medium

• Confidence: 97% confidence
• Finding: The OpenAI fallback adds third-party network exfiltration of input texts that is not reflected in the stated purpose of comparing vector stores. Because the fallback is automatic when local embedding fails, users could unintentionally transmit sensitive documents to an external API without realizing this backend does more than storage/search.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,

• Confidence: 80% confidence
• Finding: This call posts texts to a configurable Ollama endpoint, which is an external transmission from the process even if the default points to localhost. The risk is lower than a third-party SaaS call because the default target is local, but the URL is configurable and could point to a remote host, making data disclosure possible without additional safeguards.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The code transmits document text to embedding services at the call sites with no explicit user-facing warning or consent mechanism. Since embeddings are generated from raw input text, this can leak proprietary, regulated, or secret data to local or external services unexpectedly, especially through fallback behavior.

Context-Inappropriate Capability

Medium

• Confidence: 95% confidence
• Finding: A remote third-party embedding path is not necessary for the stated goal of evaluating vector stores, so it expands the trust boundary without clear justification. In this context, sending raw document text to OpenAI increases privacy and compliance risk while not being inherent to the vector-store comparison function.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 98% confidence
• Finding: The code includes an authenticated POST to OpenAI containing the input texts, which is a concrete outbound data transfer to a third party. In a skill advertised as a vector-store backend, this undisclosed behavior makes accidental data leakage more likely because callers may not anticipate remote processing.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},

• Confidence: 98% confidence
• Finding: The code includes an authenticated POST to OpenAI containing the input texts, which is a concrete outbound data transfer to a third party. In a skill advertised as a vector-store backend, this undisclosed behavior makes accidental data leakage more likely because callers may not anticipate remote processing.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,

• Confidence: 97% confidence
• Finding: The hardcoded OpenAI API endpoint evidences a built-in third-party data egress path. In this skill context, that is more dangerous because the advertised function is backend comparison, not cloud-based text processing, so users may unknowingly expose sensitive corpus contents.

External Transmission

Medium

• Category: Data Exfiltration

Content

def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,

• Confidence: 80% confidence
• Finding: The Ollama embedding path transmits input texts over HTTP to a service endpoint, which is an external transmission of potentially sensitive data. In this specific context the default target is localhost, so exposure is materially lower than an Internet service, but use of a configurable URL and plaintext HTTP can still leak data if pointed at a remote host or if local traffic is not trusted.

Static analysis

No suspicious patterns detected.