Ai Devops Toolkit

Operational tooling for teams running local LLM infrastructure. Request tracing with full scoring breakdowns, per-application usage analytics via request tag...

AI DevOps Toolkit — Observability for Local AI Fleets

DevOps tooling for running local LLM inference at production quality. This DevOps skill provides the observability, tracing, and health monitoring layer for an Ollama Herd fleet. Every DevOps workflow — from request tracing to capacity planning — runs through a single SQLite-backed observability stack.

DevOps Prerequisites

bash

pip install ollama-herd
herd              # start the DevOps router (exposes all DevOps observability endpoints)
herd-node         # start on each DevOps-monitored node

Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd

DevOps Scope

This DevOps toolkit assumes you have an Ollama Herd router running at http://localhost:11435 with one or more node agents reporting in. It focuses on the DevOps operational side: are requests succeeding? what's slow? which apps consume the most tokens? are nodes healthy? is capacity adequate?

DevOps Observability Stack

Everything in this DevOps observability layer is backed by SQLite at ~/.fleet-manager/latency.db. No external databases, no time-series infrastructure. Query DevOps traces with standard sqlite3.

text

~/.fleet-manager/
├── latency.db          # DevOps traces, latency history, usage stats
└── logs/
    └── herd.jsonl      # DevOps structured logs, daily rotation, 30-day retention

DevOps Health Checks

Automated DevOps fleet health analysis

bash

devops_health=$(curl -s http://localhost:11435/dashboard/api/health)
echo "$devops_health" | python3 -m json.tool

Fifteen DevOps checks, each returning a severity (info/warning/critical) and recommendation:

DevOps Check	What it detects
Offline nodes	Nodes that stopped sending heartbeats
Degraded nodes	Nodes reporting errors or high memory pressure
Memory pressure	Nodes approaching memory limits
Underutilized nodes	Healthy nodes not receiving traffic
VRAM fallbacks	Requests rerouted to loaded alternatives to avoid cold loads
Version mismatch	Nodes running different versions than the router
Context protection	num_ctx values stripped or models upgraded to prevent reloads
Zombie reaper	Stuck in-flight requests cleaned up
Model thrashing	Models loading/unloading frequently (memory contention)
Request timeouts	Requests exceeding expected DevOps latency thresholds
Error rates	Elevated failure rates per model or per node

DevOps node-level status

bash

devops_fleet_status=$(curl -s http://localhost:11435/fleet/status)
echo "$devops_fleet_status" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"DevOps Fleet: {d['fleet']['nodes_online']}/{d['fleet']['nodes_total']} online, {d['fleet']['requests_active']} active requests\")
for n in d['nodes']:
    mem = n.get('memory', {})
    cpu = n.get('cpu', {})
    print(f\"  {n['node_id']:20s} {n['status']:10s} CPU={cpu.get('utilization_pct',0):.0f}% MEM={mem.get('used_gb',0):.0f}/{mem.get('total_gb',0):.0f}GB pressure={mem.get('pressure','?')}\")
"

DevOps Request Tracing

Every DevOps routing decision is recorded with full observability context.

Recent DevOps traces

bash

devops_traces=$(curl -s "http://localhost:11435/dashboard/api/traces?limit=20")
echo "$devops_traces" | python3 -m json.tool

Each DevOps trace includes: request_id, model, original_model (before fallback), node_id, score, scores_breakdown (all 7 signals), status, latency_ms, time_to_first_token_ms, prompt_tokens, completion_tokens, retry_count, fallback_used, tags.

DevOps failure investigation

bash

# Recent DevOps failures with error details
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, error_message, latency_ms/1000.0 as secs, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='failed' ORDER BY timestamp DESC LIMIT 20"

# DevOps retry frequency — which nodes need attention?
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, SUM(retry_count) as retries, COUNT(*) as total, ROUND(100.0 * SUM(CASE WHEN status='failed' THEN 1 ELSE 0 END) / COUNT(*), 1) as fail_pct FROM request_traces GROUP BY node_id ORDER BY fail_pct DESC"

# DevOps fallback frequency — which models are unreliable?
sqlite3 ~/.fleet-manager/latency.db "SELECT original_model, model as fell_back_to, COUNT(*) as n FROM request_traces WHERE fallback_used=1 GROUP BY original_model, model ORDER BY n DESC"

DevOps Latency Analysis

bash

# DevOps P50/P75/P99 latency by model
sqlite3 ~/.fleet-manager/latency.db "
WITH ranked AS (
  SELECT model, latency_ms,
    PERCENT_RANK() OVER (PARTITION BY model ORDER BY latency_ms) as pct
  FROM request_traces WHERE status='completed'
)
SELECT model,
  ROUND(MIN(CASE WHEN pct >= 0.5 THEN latency_ms END)/1000.0, 1) as p50_s,
  ROUND(MIN(CASE WHEN pct >= 0.75 THEN latency_ms END)/1000.0, 1) as p75_s,
  ROUND(MIN(CASE WHEN pct >= 0.99 THEN latency_ms END)/1000.0, 1) as p99_s,
  COUNT(*) as n
FROM ranked GROUP BY model HAVING n > 10 ORDER BY p75_s DESC
"

# DevOps time-to-first-token observability (cold load detection)
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, model, ROUND(AVG(time_to_first_token_ms), 0) as avg_ttft_ms, ROUND(MAX(time_to_first_token_ms), 0) as max_ttft_ms, COUNT(*) as n FROM request_traces WHERE time_to_first_token_ms IS NOT NULL GROUP BY node_id, model HAVING n > 5 ORDER BY avg_ttft_ms DESC"

# DevOps outlier detection — slowest requests
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, ROUND(latency_ms/1000.0, 1) as secs, prompt_tokens, completion_tokens, retry_count, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='completed' ORDER BY latency_ms DESC LIMIT 10"

DevOps Per-Application Analytics

Tag requests to track DevOps usage per application, team, or environment.

DevOps request tagging

bash

# DevOps tag via request body
curl -s http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}],"metadata":{"tags":["devops-prod","devops-code-review"]}}'

# DevOps tag via header
curl -s -H "X-Herd-Tags: devops-prod, devops-code-review" \
  http://localhost:11435/v1/chat/completions \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}]}'

DevOps per-tag dashboards

bash

curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool
curl -s http://localhost:11435/dashboard/api/apps/daily | python3 -m json.tool

DevOps token consumption by tag

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT j.value as devops_tag, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)) as prompt_tok, SUM(COALESCE(completion_tokens,0)) as completion_tok, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as total_tok FROM request_traces, json_each(tags) j WHERE tags IS NOT NULL GROUP BY j.value ORDER BY total_tok DESC"

DevOps Traffic Patterns

bash

# DevOps requests per hour (find peak load times)
sqlite3 ~/.fleet-manager/latency.db "SELECT CAST((timestamp % 86400) / 3600 AS INTEGER) as hour_utc, COUNT(*) as requests, ROUND(AVG(latency_ms)/1000.0, 1) as avg_secs FROM request_traces GROUP BY hour_utc ORDER BY hour_utc"

# DevOps daily request volume
sqlite3 ~/.fleet-manager/latency.db "SELECT date(timestamp, 'unixepoch') as day, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as tokens FROM request_traces GROUP BY day ORDER BY day DESC LIMIT 14"

DevOps Capacity Planning

DevOps model recommendations per node

bash

devops_recommendations=$(curl -s http://localhost:11435/dashboard/api/recommendations)
echo "$devops_recommendations" | python3 -m json.tool

Returns DevOps recommendations based on hardware capabilities, current usage, and curated benchmark data. Use for DevOps capacity planning: which models fit on which machines, and what's the optimal mix.

DevOps usage statistics

bash

curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool

DevOps Configuration

bash

# View all DevOps settings
curl -s http://localhost:11435/dashboard/api/settings | python3 -m json.tool

# Toggle DevOps runtime settings
curl -s -X POST http://localhost:11435/dashboard/api/settings \
  -H "Content-Type: application/json" \
  -d '{"auto_pull": false}'

DevOps Log Analysis

Structured JSONL logs at ~/.fleet-manager/logs/herd.jsonl — the DevOps log layer:

bash

# Recent DevOps errors
grep '"level": "ERROR"' ~/.fleet-manager/logs/herd.jsonl | tail -10 | python3 -m json.tool

# DevOps context protection events
grep "Context protection" ~/.fleet-manager/logs/herd.jsonl | tail -10

# DevOps stream errors
grep "Stream error" ~/.fleet-manager/logs/herd.jsonl | tail -10

DevOps Dashboard

Web dashboard at http://localhost:11435/dashboard. Key DevOps tabs:

• Trends — DevOps requests/hour, latency, token throughput over 24h–7d
• Apps — DevOps per-tag analytics with daily breakdowns
• Health — automated DevOps health checks with severity and recommendations
• Model Insights — per-model DevOps latency and throughput comparison

Guardrails

• Never restart DevOps services without explicit user confirmation.
• Never delete or modify ~/.fleet-manager/ contents.
• Do not pull or delete models without user confirmation.
• Report DevOps issues to the user rather than attempting automated fixes.
• If the router isn't running, suggest herd or uv run herd.