Observability Aiops

Use this skill whenever the user needs to operate a self-hosted observability stack on Prometheus (HTTP API + PromQL), Alertmanager, or Grafana — a one-shot overview, PromQL instant/range queries, label + series metadata, scrape-target health (up/down + why) and dropped targets, recording/alerting rule health, firing/pending alerts, Alertmanager alerts + silences, Grafana dashboards/datasources/folders, three flagship analyses (firing-alert RCA, target-scrape-health, alert-noise/flap), and guarded writes (create/expire silence, create annotation, update/delete dashboard, reload Prometheus config). Always use this skill for "Prometheus", "PromQL", "Alertmanager", "Grafana", "which targets are down", "scrape failing", "why is this alert firing", "root cause this alert", "firing alerts", "silence this alert", "noisy alerts", "alert flapping", "recording rule", "alerting rule", "dashboard", "datasource health", "reload prometheus config", "TSDB cardinality" when the context is a self-hosted metrics/observability stack. Do NOT use when the target is something other than a Prometheus/Grafana observability stack (a hypervisor, storage appliance, backup product, container-orchestrator control plane, network device config, or OT/industrial equipment) — route those to the appropriate other AIops-tools skill. Hosted/SaaS monitoring suites (Datadog, New Relic, enterprise NMS) are out of scope. Preview — governed observability operations with a built-in governance harness (audit, policy, token budget, undo, risk-tiers). Mock-validated only; Prometheus and Grafana are free/open-source and trivial to run locally for a live check.

Observability AIops (preview)

Disclaimer: Community-maintained open-source project, not affiliated with, endorsed by, or sponsored by the Prometheus or Grafana projects, Grafana Labs, or the CNCF. Prometheus, Alertmanager and Grafana are trademarks of their respective owners. Source at github.com/AIops-tools/Observability-AIops under the MIT license.

Governed self-hosted observability operations — 30 MCP tools across Prometheus (HTTP API + PromQL), Alertmanager (alerts + silences), and Grafana (dashboards, datasources, folders), every one wrapped with the bundled @governed_tool harness: a local unified audit log under ~/.observability-aiops/, policy engine, token/runaway budget guard, undo-token recording, and graduated-autonomy risk tiers. One config can span the whole stack. Bearer tokens are stored encrypted (~/.observability-aiops/secrets.enc, Fernet + scrypt) — never plaintext on disk.

This is the self-hosted-observability complement to enterprise monitoring suites: it speaks the open Prometheus/Grafana APIs an SRE actually runs.

Standalone: the governance harness is bundled in the package (observability_aiops.governance) — no external skill-family dependency. Preview / mock-only: Prometheus and Grafana are free/open-source and trivial to run in a lab, so a live doctor check is easy.

What This Skill Does

Group	Platform	Tools	Count	R/W
Metrics	Prometheus	instant_query, range_query, label_values, series_metadata	4	read
Targets & status	Prometheus	list_targets, target_scrape_health, dropped_targets, prometheus_config_status, prometheus_tsdb_status	5	read
Rules	Prometheus	list_rules, rule_health	2	read
Alerts	Prometheus/Alertmanager	firing_alerts, pending_alerts, alertmanager_alerts, list_silences	4	read
Grafana	Grafana	list_dashboards, get_dashboard, list_datasources, datasource_health, list_folders	5	read
Overview + analyses	both	observability_overview + firing_alert_rca, target_scrape_health_analysis, alert_noise_and_flap_analysis	4	read
Writes	Alertmanager/Grafana/Prometheus	create_silence, expire_silence (med) · create_annotation (low) · update_dashboard (med) · delete_dashboard (high) · reload_prometheus_config (med)	6	write

The three flagship analyses are transparent heuristics that report their numbers: firing_alert_rca joins each firing alert to its rule expression and maps it to a cause + action; target_scrape_health_analysis ranks down/erroring scrape targets and classifies each lastError; alert_noise_and_flap_analysis finds noisy/duplicate alerts and recommends a dedup/rollup.

Quick Install

bash

uv tool install observability-aiops
observability-aiops init       # wizard: pick platform (prometheus/grafana) + encrypted token
observability-aiops doctor

When to Use This Skill

• Get a snapshot (overview / observability_overview): firing-alert count, scrape targets up/down, rules erroring (Prometheus) or dashboard/datasource counts (Grafana)
• Run PromQL (instant_query / range_query), enumerate label_values or series_metadata
• Check scrape health (target_scrape_health, dropped_targets) and rule health (rule_health, list_rules)
• Triage alerts: firing_alerts / pending_alerts, the Alertmanager view (alertmanager_alerts, list_silences), then firing_alert_rca to root-cause
• Reduce alert noise (alert_noise_and_flap_analysis) → group_by / inhibition / longer for
• Grafana: list_dashboards, get_dashboard, list_datasources, datasource_health, list_folders
• Governed writes: silence an alert (create_silence, time-boxed), annotate an event (create_annotation), update/delete a dashboard (dry_run + approver for delete), or hot-reload Prometheus (reload_prometheus_config)

Do NOT use when the target is not a Prometheus/Grafana observability stack — route hypervisor, storage, backup, container-orchestrator, network-device-config, or OT/industrial work to the appropriate other AIops-tools skill. Hosted/SaaS monitoring suites (Datadog, New Relic, enterprise NMS) are out of scope.

Related Skills — Skill Routing

If the user wants…	Use
Prometheus / Alertmanager / Grafana observability ops	observability-aiops (this skill)
A different platform (hypervisor, storage, backup, orchestrator, network config, OT edge)	the appropriate other AIops-tools skill
Hosted/SaaS monitoring (Datadog, New Relic, enterprise NMS)	out of scope for this tool

Common Workflows

Root-cause the firing alerts

1. firing_alerts → what is firing now, grouped by severity
2. firing_alert_rca → each firing alert joined to its rule expr with a likely cause + recommended action (advisory heuristic — verify against logs)
3. Silence the noise while you fix it: create_silence (time-boxed) on a matcher

Investigate a scrape gap

1. target_scrape_health → up/down counts + the unhealthy targets
2. target_scrape_health_analysis → down targets ranked, each lastError classified (connection refused / timeout / auth / DNS / TLS) with a fix
3. dropped_targets if a target is missing entirely (relabeled away)

Tame a noisy alert

1. alert_noise_and_flap_analysis → alertnames with many instances / exact duplicates, each with a group_by / inhibition / longer-for recommendation
2. list_silences / create_silence to quiet it now

Safely change a Grafana dashboard (reversible)

1. get_dashboard <uid> → confirm the target dashboard
2. update_dashboard <model> — it fetches + stashes the prior model and records a restore undo, or delete_dashboard <uid> --dry-run → preview
3. Re-run without --dry-run (set OBSERVABILITY_AUDIT_APPROVED_BY +

• Secure by default (v0.2.0+): with no ~/.observability-aiops/rules.yaml, high/critical operations are denied unless OBSERVABILITY_AUDIT_APPROVED_BY names an approver (set OBSERVABILITY_AUDIT_RATIONALE too). observability-aiops init seeds a starter rules.yaml; an operator-authored rules file is honoured as-is. OBSERVABILITY_AUDIT_RATIONALE for the high-risk delete) — the prior model is captured before delete so the recorded undo can recreate it

Governance & Safety

• Every tool is audited to ~/.observability-aiops/audit.db (relocatable via OBSERVABILITY_AIOPS_HOME).
• The high-risk op (delete_dashboard) can require a named approver: set OBSERVABILITY_AUDIT_APPROVED_BY and OBSERVABILITY_AUDIT_RATIONALE.
• Every write supports dry_run; the destructive one adds an approver gate.
• Silences are time-boxed (require a positive duration). Reversible writes capture the real fetched before-state and record an inverse descriptor (create_silence→expire, update/delete dashboard→restore/recreate).

References

• references/capabilities.md — full tool + platform + API-path reference
• references/cli-reference.md — CLI command reference
• references/setup-guide.md — onboarding, credentials, and connectivity