Rag Evaluator
PassAudited by ClawScan on May 1, 2026.
Overview
The skill appears to be a local Bash-based RAG logging/export tool, with no evidence of network exfiltration or destructive behavior, but users should notice its persistent local logs and some packaging/description mismatch.
This looks safe to use as a local logging CLI if you are comfortable with its data directory and exports. Verify the script before installing it as `rag-evaluator`, and avoid entering secrets or confidential prompt/evaluation data unless local retention is acceptable.
Findings (3)
Artifact-based informational review of SKILL.md, metadata, install specs, static scan signals, and capability signals. ClawScan does not execute the skill or run runtime probes.
A user may expect a Python observability SDK or official Ragaai Catalyst-style integration, but the provided artifacts behave like a local command-line log manager.
The description suggests a Python SDK, while the documented requirements and included file show a Bash CLI logging tool. This is a packaging/positioning mismatch users should notice.
description: "Python SDK for Agent AI Observability..." ... Requirements: "Bash 4+"
Treat this as a local Bash logging utility unless the publisher provides clearer provenance, installation instructions, and SDK code.
Users need to verify what executable is actually being run as `rag-evaluator` and avoid assuming the command was installed safely.
The artifacts include `scripts/script.sh` and SKILL.md instructs use of `rag-evaluator`, but no install mechanism is declared, leaving command installation/linking to the user or platform.
No install spec — this is an instruction-only skill.
Review the script before making it executable or placing it on PATH, and prefer a publisher-provided install spec or signed release.
Prompts, evaluation notes, model names, costs, or usage details entered into the tool may remain on disk and be included in exports.
The skill persistently stores user-supplied evaluation, prompt, cost, and usage entries for later search/export. This is disclosed and purpose-aligned, but it can retain sensitive project data.
All data is stored locally in `~/.local/share/rag-evaluator/` ... A unified `history.log` tracks all activity ... Export supports JSON, CSV, and plain text formats
Do not log secrets or confidential customer data unless local storage is acceptable; periodically review or delete `~/.local/share/rag-evaluator/` and exported files.