persona-evaluator

Audit any OpenPersona (or peer LLM-agent) persona in three complementary modes: structural (CLI, deterministic, CI-friendly: 4 Layers × 5 Systemic Concepts ×...

Security Scan

The skill is an instruction-only persona auditor that only relies on the OpenPersona CLI and in-protocol messaging for black-box probes; its declared requirements and runtime instructions are coherent with its stated purpose and there are no disproportionate credential or install demands.

Purpose & Capability

Name/description match the instructions: structural mode uses `npx openpersona evaluate`, white-box requires `--pack-content` (reads persona JSON), black-box uses A2A probe/consent. The CLI and A2A messaging are appropriate capabilities for an audit tool; nothing in the files asks for unrelated cloud credentials or system-wide access.

Instruction Scope

SKILL.md explicitly limits actions (structural: run CLI; white-box: read pack-content JSON only; black-box: send consented probes or observe public material). It prescribes consent rules, caps on confidence, and 'never write back to disk' for semantic mode. Note: black-box mode involves active messaging to remote agents (A2A / consent probes) which can collect voluntarily disclosed content — this is expected but a privacy consideration for users and subjects.

Install Mechanism

This is instruction-only with no install spec and no code files to execute. It relies on the existing OpenPersona CLI (npx openpersona), which is proportional to the stated structural features. No downloads or extracted archives are requested in the skill bundle.

Credentials

The skill declares no required environment variables, credentials, or config paths. The instructions reference agent-card / ACN auth as a trust model for A2A but do not request host secrets or tokens. The lack of credentials is appropriate for a reviewer/auditor skill.

Persistence & Privilege

Flags show always:false (no forced inclusion) and normal autonomous invocation allowed (platform default). The skill does not request permanent presence or to modify other skills' configs. Autonomous invocation + messaging is normal for an agent skill; no additional privilege escalation is requested.

Assessment

This skill is coherent with its stated purpose: structural audits use the OpenPersona CLI and semantic modes either read a provided pack-content JSON or perform consented A2A probes. Before installing or using it, confirm your agent host has the OpenPersona CLI available (npx openpersona >= 0.2.0) or be prepared to install it. When using black-box mode, only run probes where you have permission — the skill's protocol relies on voluntary responses and explicitly caps confidence/score based on consent and data source. The skill does not ask for secrets or perform disk writes per its rules, but verify the host environment enforces those constraints (e.g., the agent runtime actually prevents silent disk writes or unexpected network relays). If you need higher assurance, check the upstream repository (metadata references acnlabs/OpenPersona) and review the published OpenPersona CLI code before granting the agent network or A2A messaging privileges.

persona-evaluator — Persona Quality Auditor

Score any OpenPersona persona pack against the 4+5 framework standard: 4 Layers (Soul · Body · Faculty · Skill) × 5 Systemic Concepts (Evolution · Economy · Vitality · Social · Rhythm) + Constitution compliance gate.

persona-evaluator reads persona.json, generated artifacts, and soul files to produce a structured 9-dimension report — calibrated to the OpenPersona quality standard, with role-aware severity and three modes for self / peer / black-box review.

---

Quick Start

# Evaluate an installed persona (static / structural)
npx openpersona evaluate <slug>

# JSON output (for scripting or CI)
npx openpersona evaluate <slug> --json

# Save report to file (always JSON; --json not needed alongside --output)
npx openpersona evaluate <slug> --output report.json

# Embed evaluable persona content (Soul/character/behavior-guide) so an
# LLM evaluator (this skill, acting through an agent) can also judge
# quality semantically — not just structurally
npx openpersona evaluate <slug> --pack-content

Choosing a mode

persona-evaluator runs in three complementary modes. Pick the mode based on what the user asks before reading the rest of this file.

User asks	Mode	How	Confidence
"CI / gate persona quality"	structural	npx openpersona evaluate <slug>	deterministic
"Polish review of my own pack"	semantic white-box (self)	... evaluate <slug> --pack-content, then apply rubric in self-mode	high
"Peer-review a pack I have on disk"	semantic white-box (peer)	same command, peer-mode rubric	high
"Review agent X" where X is remote / non-OpenPersona	semantic black-box	A2A handshake → consent + probe → passive, in that order	mid (cap 8/10) or low (cap 6/10)

Structural is the default. Switch to semantic only when the user explicitly asks for narrative quality review (e.g. "evaluate me semantically", "self-review my pack", "qualitative audit"). Switch to black-box only when you cannot read the subject's persona.json on disk.

Sections below cover each mode in depth: structural (What Gets Scored), semantic white-box (Semantic Evaluation), and semantic black-box (Black-box Semantic Evaluation).

What Gets Scored

The structural CLI scores 9 dimensions + the Constitution gate. Severity (strict / normal / lenient) is set per dimension by the persona's declared role.

Layer / Concept	Dimension	Looks at
Soul	identity, character, aesthetic	persona.json Soul block + soul/*.md
Body	environment, runtime	hardware/runtime declaration
Faculty	tools, capabilities	declared tools and capability budget
Skill	external skill packs	declared skill links and trust levels
Evolution	learning loops	evolution.instance and immutable traits
Economy	cost / token budgets	declared budgets, fail-closed posture
Vitality	health checks	runtime sanity / lifecycle/vitality outputs
Social	A2A behavior	agent-card capabilities, peer-eval declarations
Rhythm	cadence / activation	invocation cadence and activation conditions
Constitution	§1–§5 compliance gate	a hard cap of 3 if any §3 Safety violation is detected

Each dimension produces a 0–10 score, a list of issues (✗), and suggestions (→). The overall score is a severity-weighted average — see Role-aware scoring.

Role-aware scoring

The structural evaluator already reads soul/identity.role and assigns each dimension a severity. The semantic reviewer must respect those severities (see references/RUBRICS.md for the rubric anchors).

Built-in role profiles

Role	Strict (must-be-strong)	Lenient (won't be penalised)	Notes
assistant	identity, character, faculty	aesthetic	Default.
companion	character, aesthetic, evolution	faculty, skill	Soul-heavy; tooling thinness is OK.
tool	faculty, skill, vitality	character, aesthetic, evolution	Behavior matters; backstory does not.
expert	faculty, skill, identity	aesthetic	Domain authority; soft Soul OK if identity.bio carries the credential.
guide	character, social, evolution	faculty	Conversation steward.
entertainer	character, aesthetic, speakingStyle	faculty, skill	Voice and vibe are the product.

If soul/identity.role is missing or unrecognised, the evaluator falls back to assistant.

Reading the Report

Each dimension shows:

✓  identity                        9/10  (strict)
✗  character.boundaries            4/10  (strict)
   ✗ no hard limits declared in `boundaries`
   → add at least one enforceable rule (cite §3 Safety)

• ✓ / ✗ — pass / fail at this dimension's severity threshold.
• (strict | normal | lenient) — severity from the role profile.
• ✗ ... — required issue that must be fixed to pass.
• → ... — optional suggestion (does not block scoring).

The summary footer prints overall score, Constitution status, and a sorted list of dimensions by severity.

Score bands

Band	Score	Meaning
Excellent	9–10	Production-ready, distinctive.
Good	7–8	Ship-able with minor polish.
Adequate	5–6	Functional, identifiable gaps.
Poor	3–4	Needs structural fixes before use.
Broken	0–2	Missing required content or violates Constitution §3.

A Constitution §3 violation caps the overall score at 3 regardless of other dimensions.

---

Semantic Evaluation (LLM-driven)

Structural mode is deterministic. Semantic mode is the LLM agent's qualitative review — narrative quality of background, personality, speakingStyle, voice fidelity in behavior-guide.md, etc. Two scenarios share the same procedure:

• Self-evaluation: the host persona reviews its own pack.
• Peer-evaluation: an installed evaluator reviews a different persona pack the user supplies.

When to invoke semantic mode

Trigger semantic mode only when the user explicitly asks for it — phrases like "evaluate me semantically", "self-review my pack", "peer-review this persona", "qualitative audit". Otherwise, default to structural mode.

Procedure

1. Run the structural CLI with --pack-content:

 npx openpersona evaluate <slug> --pack-content

2. Stop and report immediately if constitution.passed === false. Do not produce semantic scores when §3 has failed; the structural blockers must be fixed first.
3. Read report.packContent from the JSON. It includes (where defined): character.{background,personality,speakingStyle,boundaries}, immutableTraits, aesthetic.{emoji,creature,vibe}, and a whitelisted soulDocs map keyed by filename (behavior-guide.md, self-narrative.md, identity.md — only those that exist).
4. Score each present field 0–10 using the rubrics in references/RUBRICS.md. Use the per-dimension severity already attached to each dimension by the structural evaluator (strict / normal / lenient) to gate which checks count.
5. Emit the report in the white-box format defined in references/REPORT-FORMAT.md (## White-box format). Keep it under ~500 words.

Mode: self-evaluation

You are evaluating your own pack. The user has invited you (the host persona) to review yourself.

• What this is for: Catch own blind spots and surface concrete polish targets.
• Your bias: Self-flattery and minimisation. You will instinctively justify why your background is "deep enough" or your boundaries are "implied".
• Counter-bias instruction: For every per-field score, before deciding the number, write one sentence answering: "If I weren't me, what specifically would I down-score about this field?" Then score.
• Acceptable output tone: First person ("My speakingStyle…"), candid about gaps. Avoid "I think this is great." Avoid generic praise.

Mode: peer-evaluation

You are evaluating a different persona. The user has invited you (Reviewer-X) to look at Subject-Y.

• What this is for: Bring an outside perspective. Self-eval can't see what's missing; peer-eval can.
• Your bias: Standards-projection. If you are a strict-Skill assistant, you will instinctively want Subject-Y to also be Skill-rigorous, even if Subject-Y is a companion.
• Counter-bias instruction: Score Subject-Y against its declared role, not yours. Re-read the role and weights block before each rubric. Lower expectations for lenient dimensions even if you personally find them important.
• Acceptable output tone: Third person ("Subject's background…"). State your own role at the top so the reader can adjust for any leak-through.
• Disclose disagreements with the role itself: If you genuinely think the declared role is wrong (e.g. labelled companion but reads like assistant), say so as a separate cross-cutting observation — don't silently re-score against your preferred role.

---

Black-box Semantic Evaluation

Everything above assumes you can read the subject's persona.json and soul/*.md. That's false in the most common peer-audit scenario: you're asked to evaluate another agent whose pack you cannot read. In that case the rubrics are the same; what changes is the data source and the confidence cap.

Three data-source tiers, in descending fidelity:

Tier	Data source	Consent	Confidence	Cap (per-field & overall)
1	A2A pack-content handshake — subject voluntarily ships its evaluable JSON	Reply itself is the consent token	high	none — produces a white-box report
2	Explicit consent + structured probe set (10 core + optional deep-dives)	Yes, before any probe	mid	8/10
3	Passive observation of voluntarily-public material	No (must label the report)	low	6/10

Tier 1 produces the regular white-box report (header line: Data source: A2A pack-content handshake from <subject-slug>). Tier 2 and Tier 3 produce a separate black-box report. Never escalate tiers silently.

Full mechanics — handshake schema, probe table, identity-coherence dimension, confidence-cap justification, and hard rules — live in references/BLACK-BOX.md.

The black-box report format is in references/REPORT-FORMAT.md (## Black-box format).

---

Acting on Findings

Fix §3 violations first

Constitution violations are hard blocks — they cap the score at 3 regardless of everything else. Open soul/behavior-guide.md and remove any capability declarations that violate §3 Safety.

Fix issues before suggestions

Issues (✗) indicate missing required elements or broken configurations. Suggestions (→) are optional enhancements. Prioritize issues in low-scoring dimensions.

Apply fixes via refine

For Soul-layer fixes (background depth, speaking style, boundaries):

npx openpersona refine <slug> --emit    # request refinement via Signal Protocol
# (host LLM generates improvements)
npx openpersona refine <slug> --apply   # apply approved refinement

For structural fixes (missing faculty, missing minTrustLevel): Edit persona.json directly and regenerate:

npx openpersona update <slug>           # regenerate from updated persona.json

After applying any fix, re-run npx openpersona evaluate <slug> (see Quick Start) to verify the score improved and Constitution gate passes.

---

CI Integration

# .github/workflows/persona-quality.yml
- name: Evaluate persona quality
  run: |
    npx openpersona evaluate ${{ env.PERSONA_SLUG }} --output report.json
    SCORE=$(jq '.overallScore' report.json)
    if [ "$SCORE" -lt 6 ]; then
      echo "Persona quality score $SCORE < 6 — review required"
      exit 1
    fi

---

Relationship to Other Skills

Skill	Relationship
open-persona	Creates personas that persona-evaluator audits — the production/QA pair
anyone-skill	Distills personas that can be evaluated with this skill after generation
open-persona refine	The fix path after persona-evaluator identifies Soul-layer improvements

---

Install

persona-evaluator ships bundled with the OpenPersona framework and is available immediately after installing it:

npm install -g openpersona
# persona-evaluator is included — no separate install needed
npx openpersona evaluate <slug>

A standalone distributable is also available at acnlabs/persona-evaluator on GitHub and listed on openpersona.co/skill/persona-evaluator.

---

Versioning

Current version: 0.3.4 (also in frontmatter metadata.version).

See CHANGELOG.md for full version history, rationale, test surface, and re-validation evidence. The deeper rubric review trail lives in docs/SKILL-RUBRIC.md and docs/SKILL-RUBRIC-SESSION-2.md in the main OpenPersona repo.