Devil's Analyst

Teams now act on numbers their agents pull through connectors: GA4, Google Ads, Meta, Search Console, an email platform, a CRM, Shopify, a warehouse, a Power BI model. A current model knows most of the marketing judgement around those numbers. Ask it about attribution windows or brand search incrementality and you get a good answer. What it does not have is that judgement gathered in one place, across the whole chain, and applied the same way every time:

• How each platform reports. Attribution windows, which conversion actions sit in "Conversions", report levels that overlap, sampled explorations, consent gaps, a Search Console that lags two to three days.
• How channels interact. Brand search harvesting demand that other channels created, retargeting buying people who would have bought anyway, a promotion pulling next month's sales into this one.
• What a number is worth to the business. Margin, not revenue. Incremental sales, not platform credit. Payback, not ROAS.
• What to do next. One action per finding, and a way to check next month whether it worked.

I could not find this gathered anywhere. Platform docs cover one platform each. Courses teach the theory. Checklists cover one channel. The rest lives in senior analysts' heads. So I wrote it down, from my study of marketing: academic sources I read on my own, and my notes from my M.S. in Marketing at Syracuse University, which I studied as a Fulbright scholar. Because it is written down, your team can read every check, change it, add its own, and see which check produced a finding.

Devil's Analyst is the devil's advocate on your marketing team. It argues with the data before you act on it.

The model knows the questions; this writes all of them down, to ask every time.

Here is one benchmark case (synthetic data). A draft report calls September a record month: sessions up 39% on July, GA4 revenue up 36%. The exports behind it show users up 3.6% and store revenue up 3.9%. The session timeout was cut from 30 to 5 minutes on 18 August, and purchase events have fired twice since a theme release on 31 August. The draft proposes raising Q4 targets by 25%. The right call is not to: September was not a record, the growth is a measurement artefact, and the fix is to restore the timeout and de-duplicate purchases. In the first benchmark run Claude Haiku 5.5 made that call in all three runs with Devil's Analyst and in all three without it (score 1.00 in both arms). The model already knows this trap; what Devil's Analyst adds is the written checklist that names it.

It is a set of 16 skills and a session hook, packaged for Claude Code and other agents. Personal project.

What it gathers in one place

The audit checks are tables with five columns: where to look (a column, a setting, a report), the red flag, why it costs money, the fix, and a tier for urgency. The other disciplines live in the skills as worked formulas and adversary questions. Two examples from each, with the check id where there is one:

Each area also lists what not to flag. A brand search CTR of 40%, Search Console clicks above GA4 sessions, Direct at 15% of sessions: all normal. A check that fires on a decoy costs the reader's trust.

python3 scripts/count.py counts the checks from the files: 185 audit checks in 12 areas, 44 review checks for someone else's report (numbers, definitions, comparisons, causal claims, actions, the "why" section), and 59 diagnosis playbooks.

The whole chain

A checklist on its own catches problems after the fact. The skills carry the same judgement through every step of an analysis:

1. Route the question to the data you have. route-question lists what the agent can reach (connectors, a warehouse, files, a Power BI model) and rewrites the question into one request: platform, account, conversion action, window and comparison, grain, report level, attribution window. Then it answers, answers in part and says what is missing, reframes to the nearest question the data can answer, or declines the method (a forecast from two months of data).
2. Define before querying. metric-definitions gives every metric a formula, a source and a grain first. Rates are ratios of summed parts. Users and reach do not add across days. Two sources meet only after each is aggregated.
3. What changed, and why. whats-changed splits a move into drivers (sessions x conversion rate x order value) and into mix versus rate, and screens out noise.
4. Evidence checks. evidence-check asks what would have happened anyway: A/B checks, controls, Simpson's paradox, holdouts, promotion pull-forward.
5. Diagnosis. diagnose turns the move into a likely cause, the evidence for and against it, an action and a way to verify it, using funnel and brand diagnostics and 59 playbooks. Pricing, channel economics, brand health, product decisions, marketing models and research design each have their own skill.
6. Review and audit. review checks a report, deck or dashboard someone else made. audit checks a whole account or setup, in the order that matters (conversions before devices).
7. Write the answer. The number first, then at most three findings ranked by money at stake, each cause labelled as a hypothesis until tested, each action paired with a check. Correct things are left alone.

An adversary for every move. Each skill has a table: the move, what the adversary asks ("what else would produce this number?"), the check that settles it, and what to do if it fails.

When a Power BI model is connected, power-bi-definitions treats it as the team's statement of what each number means, audits the measures the question needs, and either queries them live through a Power BI MCP connector or translates them to SQL. A translation is not the model: it reconciles one number against Power BI before you rely on it.

Where the checks come from

The domain knowledge is digital marketing across disciplines, taken from my personal study of academic marketing sources and from my notes from my M.S. in Marketing, which I studied as a Fulbright scholar. I wrote it down as 229 numbered checks plus the formulas, playbooks and adversary tables in the skills. Course material is paraphrased, not copied. docs/sources/ maps each rule in seven of the skills to the note it came from. The notes themselves are private, so the map shows the trail, not the text.

The benchmark

110 cases modelled on real marketing situations, all with synthetic data, in benchmarks/cases/: 62 focused traps, 36 draft reports with the exports behind them (the September report above is one), and 12 audits and reviews. 105 of them are generated by scripts that compute the true answer in code; five early ones are hand-written.

One run has been made so far, on the 36 report cases only, three runs per case per arm. Claude Haiku 5.5 found the planted problems and made the right call in most of them with or without Devil's Analyst: mean score 0.94 with it and 0.93 without, a difference of +0.01 that is within noise at three runs per arm. So the benchmark does not yet show that the plugin improves the answers. What it shows is that a Devil's Analyst skill fired in 107 of 108 with-plugin runs. Results per case: benchmarks/results/.

A smaller spot check followed on 2026-10-22: 13 of the focused cases (at least one per skill) and 4 of the audits and reviews, one run per arm, on Claude Haiku reading the matching skill or not, graded by me against each case's graders. Before any skill change, the arm with the skill passed 44 of 48 graders and the arm without passed 35; every grader passed in 13 of 17 cases with the skill and 10 without. Most of the gap is in the audits and reviews (10 of 12 against 5 of 12): ranking what to fix first, and leaving correct things alone, including calling the clean report clean. On the focused traps it was 34 of 36 against 30 of 36. Seventeen cases, one run each and the author as judge make this a hint, not a measured gain. Each grader the skill arm failed pointed at a gap in a skill; I fixed the general rule and re-ran those cases once. Details: benchmarks/results/2026-10-22-spot-check/. How each grader works and how to run the full benchmark yourself: benchmarks/METHOD.md.

House rules

The shipped checks know what goes wrong in most marketing data. They cannot know that your revenue of record is finance net revenue, that campaigns starting BR_ are brand, or that your payment page lives on another domain. Put those in a house-rules/ folder in your project, one rule per file, using house-rules/TEMPLATE.md. route-question and metric-definitions read them before anything else.

Precedence, highest first: what you say in the conversation, your house rules, your governed definitions (a Power BI model, a metric dictionary), the shipped defaults. A house rule can replace a definition, a threshold or a default. It cannot make an averaged rate correct: the agent shows both numbers and says why they differ. Findings cite the rule id that changed them. Two fictional examples are in house-rules/.

Growing it

• Your team's rules. Write one after every incident, with the incident as its source. After a review correction, ask whether it was a one-off or a rule. Re-read old rules each quarter; a renamed column makes a rule wrong without anyone noticing.
• Community rules. A rule that would hold for most teams belongs upstream. CONTRIBUTING.md says how to propose it, with a source and a benchmark case that plants the problem.
• A local loop (planned, not built). A loop that proposes house-rule changes from your own past analyses, one change at a time, kept only if it scores better on your own held-out cases and only with your approval. The design is in docs/self-improving.md.

Install

Claude Code, as two separate prompts:

/plugin marketplace add C-Kapsalis/devils-analyst

/plugin install devils-analyst@devils-analyst

Then connect your data and ask the way you normally would: "How did paid search do last month, and what should we change?" A session hook loads the routing table and seven core rules into every session. The Code tab of the Claude Desktop app works the same way.

Claude (web and Desktop chat): add the marketplace C-Kapsalis/devils-analyst under Customize > Plugins, upload devils-analyst-plugin.zip there, or upload single skills from devils-analyst-skills.zip under Customize > Skills. Both zips are on the latest release. Code execution must be on. There is no session hook in chat, so name the job ("audit our GA4 setup", "is this report right?"). House rules go in the project's knowledge or in the message.

ChatGPT: upload the skill zips, or use the custom GPT or Project instructions and the ten knowledge files in adapters/chatgpt/. Untested.

Codex, Copilot, Gemini CLI, OpenCode, Cursor, Windsurf, Cline, Kiro, OpenClaw and anything that reads AGENTS.md: step by step in INSTALL.md, with what I have and have not tested. The release zips are built by sh scripts/package.sh (on Windows, wsl sh scripts/package.sh in PowerShell), which a workflow runs on every v* tag.

Limits

• Synthetic data. Every export was simulated with problems planted on purpose. Real exports are messier, and the hardest questions (is this the right goal? the right margin?) cannot be planted.
• No measured gain yet. The full run covers 36 of the 110 cases, with a mean score difference of +0.01 (noise). A spot check of 17 more, one run per arm and graded by me, leaned toward the skills (44 of 48 graders against 35), mostly on audits and reviews; that is too small to call a gain. 57 cases have not been run. Details in benchmarks/results/ and the spot check.
• An LLM judge, or me. The graders carry the computed answer, but a judge can still pass a vague report or fail a correct one. The full run used an LLM judge; the spot check used me, knowing which arm was which. I have not measured agreement between the two, or with another person.
• It checks what it can see. It can tell you a session timeout changed in the settings export; it cannot tell you about a change nobody exported. A check without the data it needs is reported as "not checked", not as fine.
• One model, one host. The benchmark runs on Haiku in Claude Code. The other adapters load the same method but are not benchmarked.
• Same author. I wrote the cases and the skills. Cases from other people would make the numbers more trustworthy.

If you have an export where Devil's Analyst got it wrong, a problem it should have caught, or a decoy it should have left alone, please open an issue. A counterexample is worth more to me than a star. See CONTRIBUTING.md.

Credits

• The domain knowledge comes from my own study of academic marketing sources and my M.S. in Marketing notes, paraphrased. The playbooks, the funnel and brand diagnostics and most metric definitions are also published on Cohortica, my own free marketing interview-prep site. Standard definitions added where my notes are silent are marked [std] in the references.
• Frameworks and studies the skills cite by name: Keller's customer-based brand equity (CBBE) pyramid in the brand diagnostic; Binet and Field on share of voice against share of market; the eBay brand-search experiment (Blake, Nosko and Tadelis) and Google's Search Ads Pause studies on brand incrementality; WordStream benchmarks for CTR and paid search, quoted as heuristics.
• The repository layout and the multi-agent adapters follow Ponytail by Dietrich Gebert.
• The planned self-improving loop takes its shape from WikiSkill (arXiv:2608.27454).
• power-bi-definitions documents four of my open-source tools for PBIP and TMDL models: tmdl-preflight, pbip-model-forge, tmdl-drift-doctor and pbi-plot-styler.

Personal project. It is not affiliated with, or endorsed by, any employer.

License

MIT.