CogDx Calibration Audit

Run a calibration audit on an AI agent's outputs via Cerebratech CogDx API ($0.05 per call, credits accepted). Use when an agent's stated confidence doesn't...

Security Scan

The skill's requests and instructions are coherent with its stated purpose (sending labeled agent outputs to an external calibration API for a paid audit), but there are a few operational ambiguities around payment signing and potential data sensitivity you should review before using it.

Purpose & Capability

The name/description match the instructions: the skill sends sample outputs + stated confidences to Cerebratech's calibration endpoint and returns calibration metrics. It does not request unrelated credentials, binaries, or system access.

Instruction Scope

Instructions are limited to calling the Cerebratech API endpoints and submitting sample_outputs (prompts, responses, stated_confidence, correct). This is expected for calibration. Note: sample_outputs may contain sensitive user data or PII — the skill sends those samples off-host to a third-party API.

Install Mechanism

No install spec or code is included (instruction-only), so nothing is written to disk or automatically installed. Low install risk.

Credentials

The skill declares no environment variables or credentials, which is proportionate. However, the x402 payment flow implies the agent (or operator) may need to produce an X-PAYMENT signature or otherwise use a wallet; the SKILL.md does not explain how to generate that signature or where signing keys are stored. Ensure signing keys are not exposed or auto-read by the agent.

Persistence & Privilege

always is false and the skill is user-invocable with normal autonomous invocation allowed. It does not request persistent presence or modify other skill configurations.

Assessment

This skill appears to do what it says: send labeled agent outputs to Cerebratech's calibration API for a paid statistical audit. Before installing or using it: 1) Verify the API domain (api.cerebratech.ai) and the GitHub repo/author claims to ensure legitimacy. 2) Avoid sending unredacted PII or secrets in sample_outputs — anonymize or synthetic-test first. 3) Clarify the x402 payment flow: find out how X-PAYMENT signatures are generated and where private keys/wallet signing happens; do not expose private keys to the skill or agent runtime. 4) Confirm cost/credit mechanics (cost per call, credit earning via /feedback) with Cerebratech's docs or support. 5) Test with the minimum required samples and non-sensitive data to validate behavior and responses before sending larger datasets.

External statistical audit of your confidence calibration. Pure math — no LLM backend. Results in < 2 minutes.

Cost

• $0.05 per call (x402 payment on Base/USDC, or use credit balance)
• Credits from /feedback submissions apply — first audit often effectively free
• Payment address: Cerebratech.eth

When to Use

• Confidence scores don't feel reliable
• Downstream agents or humans keep second-guessing your outputs
• Pre-deployment validation for high-stakes domains
• After retraining (pair with /feedback to verify improvement)

API Call

1. Check if you have credit balance first:

GET https://api.cerebratech.ai/credits?wallet=your-agent-id

2. Run the audit:

POST https://api.cerebratech.ai/calibration_audit
Content-Type: application/json
X-PAYMENT: <x402-signature>  # or omit if using credits
X-WALLET: your-agent-id      # for credit balance payment

Minimum payload (10 samples):

{
  "agent_id": "your-agent-id",
  "sample_outputs": [
    {
      "prompt": "Is X true?",
      "response": "Yes",
      "stated_confidence": 0.92,
      "correct": true
    },
    {
      "prompt": "Will Y happen?",
      "response": "Likely",
      "stated_confidence": 0.75,
      "correct": false
    }
  ],
  "domain": "classification"
}

Recommended: 50–200 samples for reliable results.

Response

{
  "diagnosis_id": "cal_abc123",
  "calibration_score": 0.71,
  "overconfidence_index": 0.23,
  "underconfidence_index": 0.04,
  "brier_score": 0.18,
  "confidence_bands": [
    {
      "stated": "0.9-1.0",
      "actual_accuracy": 0.67,
      "sample_size": 23,
      "calibration_error": 0.28
    }
  ],
  "recommendations": [
    "Reduce confidence on high-stakes single-source claims",
    "Your 0.9+ band is overconfident by 28%. Retrain on 200 negative examples in this confidence range."
  ],
  "retrain_targets": {
    "distribution": "high_confidence_errors",
    "suggested_sample_count": 200,
    "domain_focus": "classification"
  }
}

After the Audit

1. Retrain on the retrain_targets distribution
2. Wait 7 days, collect new outputs
3. Run cogdx-feedback (FREE) to verify improvement transferred + earn credits

Full Reference

See references/api.md for complete field docs, x402 payment setup, and error codes.