Inference Aiops

Use this skill whenever the user needs to operate a GPU inference cluster — vLLM (OpenAI API + Prometheus /metrics) and Ray Serve / Ray Jobs (Ray dashboard): a one-shot cluster overview (deployments + total replicas + queue backpressure), request metrics (TTFT / TPOT / e2e latency + token totals), queue depth, KV-cache stats (utilisation, prefix-cache hit rate, preemptions), the flagship latency root-cause analysis (diagnose_latency_spike) and low-utilisation RCA, Ray Serve autoscaling and scaling (scale up/down, scale-to-zero, drain a replica), LoRA load/unload, base-model hot-swap, deploy/undeploy/redeploy, prefix-aware routing, GPU utilisation, Ray jobs, and cost per million tokens. Always use this skill for "why is inference slow", "TTFT spike", "latency spike", "GPU underutilised", "scale down the deployment", "scale to zero", "drain a replica before a reboot", "hot-swap the base model", "load a LoRA adapter", "KV cache pressure", "prefix cache hit rate", "queue backpressure", "autoscale config", or "cost per token" when the context is a vLLM / Ray Serve inference cluster. Do NOT use for non-inference infrastructure (hypervisors, storage appliances, backup products, general container/cluster workloads, network devices, or OT/industrial equipment) — those belong to other AIops-tools; this skill is scoped to GPU inference serving (vLLM + Ray). Preview — governed vLLM + Ray inference operations with a built-in governance harness (audit, policy, token budget, undo, risk-tiers). Mock-validated only, not yet verified against a live cluster.

Inference AIops (preview)

Disclaimer: Community-maintained open-source project, not affiliated with, endorsed by, or sponsored by the vLLM or Ray projects or any inference-serving vendor. Product and trademark names belong to their owners. Source at github.com/AIops-tools/Inference-AIops under the MIT license.

Governed GPU-inference operations for vLLM (OpenAI API + Prometheus /metrics) and Ray Serve / Ray Jobs (Ray dashboard) — 30 MCP tools, every one wrapped with the bundled @governed_tool harness: a local unified audit log under ~/.inference-aiops/, policy engine, token/runaway budget guard, undo-token recording, and graduated-autonomy risk tiers. The flagship diagnose_latency_spike folds queue depth + KV-cache pressure + prefix-cache locality into a ranked cause and the specific knob to turn. vLLM's Prometheus /metrics is parsed directly — no Prometheus server required.

Standalone: the governance harness is bundled in the package (inference_aiops.governance) — no external skill-family dependency. Preview / mock-only: not yet validated against a live cluster. A bearer token is optional (many stacks run open).

What This Skill Does

Group	Tools	Count	Read or Write
Metrics & RCA	request metrics, queue depth, KV-cache stats, diagnose latency spike, diagnose low utilisation	5	5 read
Ray Serve (read)	deployment list, deployment status, replica list, autoscale config get	4	4 read
Ray Serve (write)	scale up (med), scale down (high), scale-to-zero (high), autoscale config update (med), drain replica (high)	5	5 write
Models / vLLM	model list, model info, LoRA load (med), LoRA unload (high), base hot-swap (high)	5	2 read / 3 write
Ray cluster / jobs / GPU	cluster resources, dashboard status, job list, GPU utilisation, job cancel (med), replica restart (high)	6	4 read / 2 write
Deploy lifecycle	deploy (med), undeploy (high), redeploy (high), routing policy update (med)	4	4 write
Cost	cost per token	1	1 read

16 read, 14 write. The high-risk writes support dry_run + double-confirm; reversible writes record an undo descriptor.

Quick Install

bash

uv tool install inference-aiops
inference-aiops init       # interactive wizard: host + ray_port + vllm_port + scheme (token optional)
inference-aiops doctor     # probes the Ray dashboard and vLLM independently

When to Use This Skill

• Triage a cluster (overview): Serve deployments, total replicas, queue backpressure
• Diagnose slow inference (metrics diagnose / diagnose_latency_spike): rank the cause (queue depth vs KV-cache preemption vs prefix-cache locality) and get the knob to turn
• Find idle GPUs and over-provisioned replicas (diagnose_low_utilization)
• Scale a Ray Serve deployment up/down, scale-to-zero to stop cost bleed, or update autoscale bounds
• Drain a replica gracefully before a node reboot (finishes in-flight requests)
• Load/unload a LoRA adapter; hot-swap a base model (Sleep-Mode swap, captures the prior model)
• Inspect GPU utilisation per node, list/cancel Ray jobs, restart a stuck replica
• Compute cost per million tokens from throughput × GPU $/hr

Do NOT use for non-inference infrastructure (hypervisors, storage appliances, backup products, general container workloads, network devices, or OT/industrial equipment) — those belong to other AIops-tools. This skill is scoped to GPU inference serving (vLLM + Ray).

Related Skills — Skill Routing

If the user wants…	Use
vLLM / Ray Serve inference: latency RCA, autoscale, drain, LoRA, cost/token	inference-aiops (this skill)
Any non-inference infrastructure (hypervisor, storage, backup, general clusters, network, OT)	the appropriate other AIops-tools line

Common Workflows

"Why is inference slow?" (flagship RCA)

1. inference-aiops metrics diagnose (or the diagnose_latency_spike tool) → a ranked cause with numbers: is waiting queue depth high (backpressure)? Are there KV-cache preemptions? Has the prefix-cache hit rate dropped (routing lost locality)?
2. Act on the named knob: add replicas (scale_replicas_up), raise the batch cap via autoscale_config_update, or fix locality with routing_policy_update (prefix-aware / session-affinity).
3. Re-check with request_metrics (TTFT / TPOT / e2e) and queue_depth.

Safely scale down / scale-to-zero a prod deployment

1. Set an approver: export INFERENCE_AUDIT_APPROVED_BY=you INFERENCE_AUDIT_RATIONALE="off-peak cost save".
2. inference-aiops serve scale-to-zero <app> <deployment> --dry-run → preview the call (double-confirm required). scale_to_zero stops the cost bleed but strands ingress — confirm that's intended.
3. Re-run without --dry-run; the undo descriptor captures the prior replica count so you can restore it with scale_replicas_up.

Graceful drain before a node reboot

1. drain_replica <app> <deployment> <replica_id> --dry-run (high risk) → previews; the drain finishes in-flight requests before removing the replica.
2. Confirm, drain, then reboot the node. (Multi-node drain is unverified in preview.)

Hot-swap a base model

1. model_hot_swap <new_model> (high risk, dry-run first) → a Sleep-Mode base swap that captures the prior model into an undo descriptor.
2. Verify with model_info / request_metrics; replay the undo to roll back.

Cost per million tokens

cost_per_token derives a deterministic $/1M tokens from measured throughput × your GPU $/hr — useful for sizing replicas or justifying a scale-to-zero.

Governance & Safety

• Every tool is audited to ~/.inference-aiops/audit.db (relocatable via INFERENCE_AIOPS_HOME).
• High-risk ops can require a named approver: set INFERENCE_AUDIT_APPROVED_BY and INFERENCE_AUDIT_RATIONALE.
• The fragile prod writes support --dry-run and double confirmation at the CLI.
• Reversible writes (scale, autoscale-config, routing, hot-swap, LoRA load) record an inverse descriptor.

References

• references/capabilities.md — full tool → backend → endpoint → returns reference
• references/cli-reference.md — CLI command reference
• references/setup-guide.md — onboarding, optional token, and connectivity