Rag Evaluator
AdvisoryAudited by Static analysis on Apr 30, 2026.
Overview
No suspicious patterns detected.
Findings (0)
Artifact-based informational review of SKILL.md, metadata, install specs, static scan signals, and capability signals. ClawScan does not execute the skill or run runtime probes.
A user may expect a Python observability SDK or official Ragaai Catalyst-style integration, but the provided artifacts behave like a local command-line log manager.
The description suggests a Python SDK, while the documented requirements and included file show a Bash CLI logging tool. This is a packaging/positioning mismatch users should notice.
description: "Python SDK for Agent AI Observability..." ... Requirements: "Bash 4+"
Treat this as a local Bash logging utility unless the publisher provides clearer provenance, installation instructions, and SDK code.
Users need to verify what executable is actually being run as `rag-evaluator` and avoid assuming the command was installed safely.
The artifacts include `scripts/script.sh` and SKILL.md instructs use of `rag-evaluator`, but no install mechanism is declared, leaving command installation/linking to the user or platform.
No install spec — this is an instruction-only skill.
Review the script before making it executable or placing it on PATH, and prefer a publisher-provided install spec or signed release.
Prompts, evaluation notes, model names, costs, or usage details entered into the tool may remain on disk and be included in exports.
The skill persistently stores user-supplied evaluation, prompt, cost, and usage entries for later search/export. This is disclosed and purpose-aligned, but it can retain sensitive project data.
All data is stored locally in `~/.local/share/rag-evaluator/` ... A unified `history.log` tracks all activity ... Export supports JSON, CSV, and plain text formats
Do not log secrets or confidential customer data unless local storage is acceptable; periodically review or delete `~/.local/share/rag-evaluator/` and exported files.