edge-cpu-gguf-tuner

Evidence-first, offline tuning of llama.cpp GGUF inference on CPU and constrained edge hosts. Inspects CPU topology, renders compatible benchmark plans, optionally runs an explicitly supplied local llama-bench, ranks pp/tg/pg results with variance-aware confidence, verifies output gates, and renders (never executes) a measured command. It does not install llama.cpp, download models, call APIs, or claim universal defaults.

Use this skill when a user wants to improve llama.cpp GGUF inference on a CPU-only or constrained machine, especially when thread count, batch size, flash attention, KV-cache type, context depth, model compatibility, or token-generation throughput is in question.

This is an evidence-first local measurement workflow, not a collection of universal tuning folklore. The old registry artifact contained results from one 2-vCPU/2-GB environment; those numbers are not defaults here. A recommendation is valid only for the exact host, model, llama.cpp binary/build, benchmark metric, and conditions recorded in its report.

Non-negotiable operating contract

1. Offline and no-install: never install llama.cpp, download a model, call a cloud provider, or modify a source checkout. The tool uses Python's standard library only.
2. Explicit assets: inspect and plan without assets. Run a benchmark only after the user supplies a local llama-bench executable and a local .gguf path. Do not search for or select a model implicitly.
3. No shell execution: use the entrypoint or python3 scripts/edge_cpu_tuner.py; the runner invokes an argument vector with shell=False, a reduced environment, stdin closed, and a timeout. Never paste model paths into a shell command unquoted.
4. No invented results: a plan is not a measurement. If the run fails, returns no structured records, or has no clear winner, report that state and do not fill in a number.
5. Least privilege: run the user-owned binary as a non-privileged user. A GGUF file and its parser are user-selected binary inputs; path validation is not a sandbox. Keep llama.cpp current and protect reports, which can contain paths and diagnostics.
6. Compatibility before speed: check the exact binary's -h output before using a rendered command. Current upstream documentation uses llama cli; other builds expose llama-cli or legacy llama-completion. Never assert that one name works everywhere.

Fast workflow

From the installed skill directory:

bash

# 1. Read-only host and binary discovery
sh bin/edge-cpu-tuner inspect --json

# 2. Offline plan; no executable or model is opened
sh bin/edge-cpu-tuner plan --sweep all \
  --model /path/to/model.gguf --binary /path/to/llama-bench --json

# 3. Explicit local measurement; choose one metric and keep reports
sh bin/edge-cpu-tuner bench \
  --model /path/to/model.gguf \
  --binary /path/to/llama-bench \
  --sweep threads --repetitions 3 --metric tg \
  --out reports/threads.json --json

# 4. Re-rank an existing report without rerunning inference
sh bin/edge-cpu-tuner recommend --report reports/threads.json --metric tg --json

# 5. Render a candidate command; this never launches inference
sh bin/edge-cpu-tuner deploy --report reports/threads.json \
  --cli /path/to/llama-cli --prompt 'short user prompt' --tokens 128 --json

Run separate bench invocations for threads, batch, flash, kv, and context when attributing causality. all is convenient but still emits separate configurations; it is not permission to compare unrelated changes. Use --keep-going only when partial failures are useful. --allowed-root can require the model to be below a chosen directory.

The runner's current benchmark argv uses documented llama-bench options: -m, -p, -n, -t, -b, -ub, -fa, -ctk, -ctv, -d, -r, and -o json. If the exact binary rejects an option, stop and inspect its help or use a compatible binary; do not silently fall back to a guessed flag.

How to reason about results

• Select tg for decode/token-generation throughput, pp for prompt processing, or pg for prompt-plus-generation. Do not call one metric “overall speed.”
• Upstream llama-bench reports average and standard-deviation tokens/sec, and its timings exclude tokenization and sampling. Explain this whenever a user asks about end-to-end latency.
• The default thread plan starts at one, doubles toward detected physical cores, and compares logical cores when they differ. This follows upstream troubleshooting guidance but is not a winning default.
• Keep warm-up, thermal state, CPU governor/frequency, background load, model context depth, and repetition count visible in the report. On a small box, sustained thermal throttling can reverse a short burst result.
• Prefer a measured configuration only when its margin and variance justify it. The tool labels a result winner, provisional_winner, no_clear_winner, or single_observation; low confidence means “keep the baseline or rerun,” not “guess.”
• Quantization, KV-cache type, flash attention, batch size, context depth, and architecture interact. Never port a result from one model, quantization, OS, compiler, or llama.cpp commit to another without a new run.
• A benchmark winner may use more memory or have worse latency/quality. Check memory headroom and run a user-approved end-to-end/quality comparison before deployment.

Quality and deployment gate

deploy emits a command object with executed: false, the selected argv, a shell-rendered preview, and warnings. It does not alter files or launch the model. Confirm the exact CLI's help first, then run it yourself if desired.

For a deterministic output check, capture the baseline and candidate outputs using the exact same prompt, model, seed/temperature settings supported by that CLI, and token limit. Compare them locally:

bash

sh bin/edge-cpu-tuner verify-output \
  --baseline /tmp/baseline.txt --candidate /tmp/candidate.txt \
  --mode bytes --json

identical: false means reject the candidate for an identity-gated change; it does not prove which output is semantically better. A byte-identical output is not a proof of correctness, safety, or quality. If the CLI is nondeterministic, document that and use a human/automated quality test rather than pretending identity is meaningful.

Machine-readable contract

Every JSON response has a versioned schema. Important schemas are:

• edge-cpu-gguf-tuner.host.v1 — host topology and discovered binaries.
• edge-cpu-gguf-tuner.plan.v1 — offline configurations and argv arrays.
• edge-cpu-gguf-tuner.report.v1 — exact host/model/binary metadata, attempts, parsed records, warnings, ranked metrics, and recommendation.
• edge-cpu-gguf-tuner.command.v1 — an unexecuted deployment argv and compatibility warnings.
• edge-cpu-gguf-tuner.quality-gate.v1 — hashes, sizes, identity decision, and explicit rejection/pass state.
• edge-cpu-gguf-tuner.error.v1 — safe error text and exit code.

Agents should preserve these fields when summarizing:

text

status: plan | measured | failed | no_clear_winner
host: physical/logical CPUs, memory, architecture
model_and_build: exact path/size, binary path/version, optional model hash
metric: pp | tg | pg
recommendation: configuration, tokens_per_second, standard deviation, confidence
provenance: report path plus source/evidence scope
warnings: compatibility, thermal/noise, memory, quality, or failed attempts
next_action: one explicit rerun/verification step, or “none”

Be concise by default. Do not repeat the entire report when a compact JSON object or table answers the question; expand only when the user asks for methodology or troubleshooting.

Compatibility and build evidence

The evidence ledger in references.json records retrieval date, source URLs, SHA-256 digests, claim scope, and whether a statement is upstream documentation or a local test. Current upstream references establish that:

• the README quick start uses unified llama cli, while upstream build examples also show llama-cli;
• llama-bench supports pp/tg/pg, repetitions, JSON/JSONL output, context depth, separate batch/ubatch, and -fa on|off|auto;
• benchmark timings exclude tokenization and sampling;
• excessive thread counts can oversaturate a CPU, so measuring from one thread upward is safer than blindly using every logical CPU;
• a baseline CPU build is cmake -B build followed by cmake --build build --config Release.

Those sources do not authorize this skill to build/install anything, and they do not prove a performance result on the user's hardware. The historical registry copy is retained only in the private workspace's base/ directory, not as an authority for this skill.

Self-improvement without self-modification

The safe feedback loop is evidence accumulation, not autonomous code editing:

1. save reports with the exact host/model/build metadata;
2. repeat after a controlled change or environment change;
3. use recommend to re-rank records and compare the new report to the old one;
4. update the user's chosen runtime command only after the quality/memory gate;
5. rerun after a llama.cpp update, model/quantization change, hardware migration, or thermal/power change.

The skill never rewrites its own source, fetches “better” models, or learns from unverified model-generated suggestions. The deterministic regression suite is sh selftest.sh; its fake benchmark is only a parser/control-flow test and must never be reported as real throughput.

Failure and exit semantics

• 0: successful inspection/plan/render/quality pass, or a benchmark with structured records (even if warnings are present).
• 1: benchmark produced no records or partial results; inspect warnings and rerun deliberately.
• 2: invalid user input or missing explicit asset.
• 124: subprocess timeout; stop and inspect model/binary/runtime conditions.
• 127: executable could not be launched.
• 130: user interruption.

References

• references.json — machine-readable evidence ledger and source hashes.
• README.md — full operational guide, threat model, schemas, and test checklist.
• scripts/edge_cpu_tuner.py — stdlib-only implementation.
• scripts/selftest.py / selftest.sh — deterministic static/behavioral smoke tests.