Memory Health Probe

QMD memory system telemetry — measure index health, BM25 retrieval quality, coverage maps, and trend analysis. Use when running QMD memory backend and need d...

Security Scan

The skill's code and instructions are coherent with its stated purpose (local QMD telemetry); it reads local QMD state and gateway logs and posts telemetry to a local Langfuse endpoint, and it does not request unrelated credentials or external installs.

Purpose & Capability

Name/description (QMD memory telemetry) matches the actual actions: it runs a qmd binary to collect index and BM25 metrics, reads OpenClaw gateway logs, computes coverage and trends, and stores snapshots. Required binaries (python3 + qmd) are appropriate for this purpose. No unrelated third-party credentials or services are requested.

Instruction Scope

The runtime instructions and included script stay within telemetry scope: querying qmd, scanning gateway logs (~/.openclaw/logs/...), computing BM25 hit/score and coverage, and storing snapshots. The script posts telemetry to LANGFUSE_URL (default http://localhost:3100). It can include qmd:// URIs and other metadata in payloads; it does not read or transmit raw document contents. Users should note it reads files under the user's home directory (gateway logs) which can contain sensitive metadata or error stacks.

Install Mechanism

Instruction-only skill with a bundled Python script; there is no install spec and nothing is downloaded or written to system locations by an installer. Risk from install mechanism is low.

Credentials

The skill declares no required environment variables or secrets and the script uses hardcoded local Langfuse credentials/URL. It does read local log files (gateway logs) and QMD outputs which may contain sensitive metadata; while this is proportional to telemetry, users should be aware that metadata (qmd:// paths, query hits) is sent to the configured Langfuse endpoint. If LANGFUSE_URL were changed to an external host, telemetry could be exfiltrated — verify it remains local before use.

Persistence & Privilege

The skill is not always-enabled and does not request elevated or system-wide persistence. It writes snapshots into its local data directory only and does not modify other skills or system configs.

Assessment

This skill appears to do exactly what it says: local telemetry for a QMD memory backend. Before installing or running it, do the following: (1) Review the script (scripts/memory_probe.py) yourself or run it in --dry-run to see exactly what it prints; (2) Confirm LANGFUSE_URL is localhost and not pointed at an external host; if you don't run Langfuse locally, treat the network calls as failing and inspect what would be sent; (3) Be aware it reads ~/.openclaw/logs/gateway.log and gateway.err.log — those logs can contain sensitive metadata, so run it in a trusted environment or sanitize logs if needed; (4) If you don't want qmd:// path or other metadata in telemetry, remove or sanitize those fields before sending; (5) Optionally change hardcoded Langfuse keys/URL to environment variables so the agent doesn't ship static credentials. Overall the skill is coherent, but audit data flows you consider sensitive before allowing it to run regularly.

Telemetry for QMD memory systems. Measures index health, retrieval quality, and coverage to catch degradation before it affects agent performance.

Metrics Captured

1. Index health — file count, chunk count, index size, staleness
2. BM25 quality — hit rate + score distribution over canonical queries
3. Gateway events — session-memory saves, QMD armed events
4. Coverage map — which collections are hit for which query categories

Usage

python3 scripts/memory_probe.py              # Run probe, log to Langfuse
python3 scripts/memory_probe.py --dry-run    # Print results only
python3 scripts/memory_probe.py --trend      # Show trend over stored snapshots

Files

• scripts/memory_probe.py — Probe script with Langfuse integration