{"skill":{"slug":"skylv-prompt-evaluation","displayName":"Skylv Prompt Evaluation","summary":"Evaluate and benchmark AI prompts for quality, consistency, and performance. Triggers: prompt evaluation, prompt testing, prompt quality, prompt benchmark, p...","description":"---\nname: prompt-evaluation\nslug: prompt-evaluation\nversion: 1.0.0\ndescription: \"Evaluate and benchmark AI prompts for quality, consistency, and performance. Triggers: prompt evaluation, prompt testing, prompt quality, prompt benchmark, prompt optimization.\"\nauthor: SKY-lv\nlicense: MIT-0\ntags: [evaluation, prompts, quality]\nkeywords: [prompt evaluation, prompt testing, prompt quality, prompt benchmark, prompt optimization, LLM evaluation, prompt engineering, prompt scoring]\ntriggers: prompt-evaluation\n---\n\n# Prompt Evaluation\n\nEvaluate and benchmark AI prompts for quality, consistency, and performance. Score, compare, and optimize your prompts systematically.\n\n## Overview\n\nA prompt evaluation framework that helps agents measure prompt quality across multiple dimensions: clarity, specificity, robustness, cost-efficiency, and output consistency. Compare prompt variants and find the optimal version.\n\n## Capabilities\n\n### 1. Quality Scoring\n\n```bash\nnode evaluate.js score --prompt \"Summarize the article\" --dimensions clarity,specificity,robustness\nnode evaluate.js score --prompt-file ./prompts/ --output scores.json\n```\n\nScores prompts on clarity (0-10), specificity (0-10), robustness (0-10), and cost-efficiency (0-10).\n\n### 2. A/B Comparison\n\n```bash\nnode evaluate.js compare --prompt-a \"Summarize\" --prompt-b \"Write a 3-bullet summary\" --trials 50\nnode evaluate.js compare --config ab-test-config.json\n```\n\nRun statistical A/B tests between prompt variants with significance analysis.\n\n### 3. Consistency Check\n\n```bash\nnode evaluate.js consistency --prompt \"Translate to French\" --runs 100 --variance-threshold 0.15\nnode evaluate.js consistency --temperature 0.7 --top-p 0.9\n```\n\nMeasures output consistency across multiple runs to find the most stable prompts.\n\n### 4. Regression Testing\n\n```bash\nnode evaluate.js regression --baseline v1.0 --current v1.1 --test-suite golden-set.jsonl\nnode evaluate.js regression --fail-on-degradation 5%\n```\n\nDetects quality regressions between prompt versions using golden test sets.\n\n### 5. Cost Analysis\n\n```bash\nnode evaluate.js cost --prompt \"Long prompt...\" --model gpt-4 --estimate-tokens\nnode evaluate.js cost --compare-prompts --output cost-report.csv\n```\n\nEstimates token usage and costs for different prompt variants and models.\n\n## Configuration\n\n```json\n{\n  \"evaluation\": {\n    \"dimensions\": [\"clarity\", \"specificity\", \"robustness\", \"cost\"],\n    \"scoringModel\": \"gpt-4\",\n    \"abTest\": {\n      \"trials\": 50,\n      \"significanceLevel\": 0.05\n    },\n    \"consistency\": {\n      \"runs\": 100,\n      \"varianceThreshold\": 0.15\n    },\n    \"regression\": {\n      \"degradationThreshold\": \"5%\",\n      \"goldenSet\": \"./golden-set.jsonl\"\n    }\n  }\n}\n```\n\n## Use Cases\n\n- Prompt Engineering: Systematically improve prompt quality\n- Quality Assurance: Ensure prompts meet quality standards before production\n- Cost Optimization: Find prompts that achieve goals with fewer tokens\n- Version Control: Track prompt quality across versions\n- Agent Tuning: Optimize agent system prompts for consistency\n","tags":{"latest":"1.0.0"},"stats":{"comments":0,"downloads":372,"installsAllTime":14,"installsCurrent":1,"stars":0,"versions":1},"createdAt":1777783748240,"updatedAt":1778492833926},"latestVersion":{"version":"1.0.0","createdAt":1777783748240,"changelog":"Initial release of the prompt-evaluation skill.\n\n- Evaluate and benchmark AI prompts for clarity, specificity, robustness, and cost-efficiency.\n- Score prompts, compare variants with A/B tests, and measure output consistency.\n- Run regression testing to detect quality changes across prompt versions.\n- Estimate and compare token usage and cost for different prompts and models.\n- Designed for prompt engineering, quality assurance, and cost optimization.","license":"MIT-0"},"metadata":{"setup":[],"os":null,"systems":null},"owner":{"handle":"sky-lv","userId":"s17fgkeb63szvtadtmm753m0gd84e4vz","displayName":"SKY-lv","image":"https://avatars.githubusercontent.com/u/259750852?v=4"},"moderation":null}