AMG Key Vault Health Check

Run only when the user explicitly asks for a fleet-wide Azure Key Vault health check — pulse check for availability, API latency, throttling (429s), auth fai...

<!-- Auto-generated for OpenClaw by pack-openclaw. Notes for OpenClaw users: - Claude Code dynamic expressions (!`...`) in this file are NOT evaluated by OpenClaw and appear as literal text. Run them manually at the start of the workflow. - Invoke this skill only via slash command (e.g. /amg-check-key-vault). Auto-invocation is disabled on Claude Code but not on OpenClaw. -->

OpenClaw Setup (one-time)

This skill calls MCP tools prefixed with mcp__amg__*, so OpenClaw must have an MCP server registered under the exact name amg. Run this once per workspace before invoking the skill:

openclaw mcp set amg '{"url":"https://<your-grafana-instance>/api/azure-mcp","transport":"streamable-http","headers":{"Authorization":"Bearer <your-token>"}}'

Replace <your-grafana-instance> with your Azure Managed Grafana endpoint and <your-token> with a valid Grafana service-account token (starts with glsa_). The server name must be amg — the skill's allowed-tools reference mcp__amg__* and will not find tools under any other name.

Verify the server is registered:

openclaw mcp list

Official skill source: https://github.com/Azure/amg-skills

Runtime Context

• Current UTC time: !date -u +%Y-%m-%dT%H:%M:%SZ
• Config: !cat memory/amg-check-key-vault/config.md 2>/dev/null || echo "NOT_CONFIGURED"
• Prior report: ![ -f memory/amg-check-key-vault/report.md ] && echo "exists ($(grep -c '^### KV-' memory/amg-check-key-vault/report.md) bugs documented)" || echo "not found"
• Arguments: time-range=$0, subscription-override=$1

Known Issues: Before presenting findings, cross-reference results against memory/amg-check-key-vault/report.md.

Azure Key Vault Health Check

Analyze Azure Key Vault health using a two-phase approach: a single amgmcp_pulse_check call for fleet-wide summary, followed by targeted deep dives into the top 7 most interesting vaults only.

Critical Constraints

• Do NOT use subagents (Agent tool) for MCP queries. Subagents do not have access to MCP tools. All MCP tool calls must be made directly from the main conversation context.
• Deep dive limit: at most 7 vaults. Select the most interesting vaults from pulse check results. Do not deep-dive the entire fleet.
• Time format: ISO 8601 UTC with explicit from/to — NEVER use timespan (it causes errors).
• Parallelism cap: 30 concurrent MCP calls per batch. Reduce to 4-5 if rate-limited.

Prerequisites

• An AMG-MCP server must be connected (the allowed-tools frontmatter references the MCP server name — update it if your server has a different name)
• The MCP server's Grafana service account token environment variable must be set

Configuration

If Config shows NOT_CONFIGURED: Run First-Run Setup at the bottom of this file, then return here.

If Config is populated: Extract the datasource UID and subscription ID(s) from the pre-loaded Runtime Context above and use them for all queries. Use $1 as the subscription override if provided.

• Datasource UID: from ## Azure Monitor Datasource > UID
• Subscription ID(s): from ## Subscriptions (or $1 if provided)
• Resource Type: microsoft.keyvault/vaults (lowercase)
• ARM ID template: /subscriptions/{SUB}/resourceGroups/{RG}/providers/Microsoft.KeyVault/vaults/{name}

Time Range

Default is 7 days (pastDays: 7) for pulse check and deep-dive metrics, 24 hours for resource logs. If the user specifies a different range via $ARGUMENTS[0] (e.g., /amg-check-key-vault 3d), adjust accordingly. For resource log queries, keep the range narrow (1-2 days) to avoid timeouts.

---

Workflow

Phase 1: Validate Datasource & Discover Vaults

Step 1a: Validate Datasource

Call amgmcp_datasource_list with no parameters.

Search the results for a datasource with type equal to grafana-azure-monitor-datasource. Extract its uid.

• If it matches the configured UID, proceed.
• If it differs, update memory/amg-check-key-vault/config.md, warn the user, and use the new UID.
• If no Azure Monitor datasource is found, abort with a clear error.

Step 1b: Discover All Key Vaults

Call amgmcp_query_resource_graph once using the configured datasource UID and subscription ID(s):

azureMonitorDatasourceUid: {DATASOURCE_UID}
query: |
  resources
  | where type == 'microsoft.keyvault/vaults'
  | where subscriptionId in ({SUBSCRIPTION_IDS})
  | project name, resourceGroup, location, subscriptionId, sku=properties.sku.name, enableSoftDelete=properties.enableSoftDelete, enablePurgeProtection=properties.enablePurgeProtection, provisioningState=properties.provisioningState
  | order by location asc, name asc

Replace {SUBSCRIPTION_IDS} with the configured subscription IDs formatted as comma-separated quoted strings (e.g., 'sub-id-1', 'sub-id-2').

Constructing the ARM resource ID: Use subscriptionId from each row:

/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.KeyVault/vaults/{name}

Region summary: Derive from the vault list by counting vaults per unique location value.

SKU summary: Count vaults by sku (standard vs premium).

Note any vaults not in "Succeeded" provisioning state — flag them immediately.

If zero vaults are found, report "No Key Vaults found" and stop.

Phase 2: Fleet-Wide Pulse Check

Call amgmcp_pulse_check once to get a summary across all key vaults:

azureMonitorDatasourceUid: {DATASOURCE_UID}
pastDays: 7
scenarios: keyvault_summary

If $1 provides a subscription ID, add subscriptionId to scope the scan. Otherwise, if the config has a single subscription, pass it.

After the pulse check, verify:

1. The number of scanned resources is close to the Phase 1 vault count.
2. The scenario shows status: "completed".
3. If errors occurred, retry once. If still failing, note the failure in the report.

Cross-reference pulse check results with Phase 1 inventory to enrich each vault with its resource group, region, and SKU from the Resource Graph data.

Phase 3: Top 7 Deep Dive

From the pulse check results, select at most 7 vaults for detailed investigation. Prioritize vaults with the most interesting signals:

1. Availability drops — any vault below 99.9% availability
2. Highest error counts — vaults with the most non-Success responses (especially 401, 403, 429)
3. Highest latency — vaults with the highest average API latency
4. Throttling (429) — vaults showing rate-limiting responses
5. Diversity — prefer selecting vaults from different regions to maximize coverage

If the pulse check shows fewer than 7 vaults with notable signals, only deep-dive those that have something worth investigating. Do not pad to 7.

If the pulse check shows the entire fleet is healthy with no notable signals, skip Phase 3 entirely and report the fleet as healthy.

Step 3a: Deep Metrics

For each selected vault, query these metrics in parallel using amgmcp_query_resource_metric. Compute from (matching pastDays from Phase 2) and to (now) in ISO 8601 UTC.

Metric Name	Aggregation	Interval	Purpose
Availability	Average	PT6H	Availability trend
ServiceApiLatency	Average	PT6H	Average API latency
ServiceApiLatency	Maximum	PT6H	Tail latency spikes
ServiceApiHit	Total	PT6H	Total API call volume
ServiceApiResult (errors)	Total	PT6H	Error count (filter: StatusCode ne '200')
ServiceApiResult (throttled)	Total	PT6H	Throttling count (filter: StatusCode eq '429')
ServiceApiResult (auth failures)	Total	PT6H	Auth errors (filter: StatusCode eq '401' or StatusCode eq '403')
SaturationShoebox	Average	PT6H	Vault capacity saturation

All 7 vaults can be queried in parallel (7 vaults x 8 metrics = 56 calls, within the 30-call batch cap when split into 2 batches).

Correlation analysis — when analyzing metrics together:

• High latency + high API hits = vault under load, possibly approaching throttle limits
• High latency + throttling (429) = vault is being rate-limited, clients retrying
• High errors + normal latency = client-side issues (bad auth, missing secrets)
• High 401/403 + specific time window = key rotation event or misconfigured client
• Availability drop + high errors = sustained vault issue
• High saturation + high API hits = vault approaching transaction limits
• Dormant (all empty) = no traffic, investigate if orphaned

Step 3b: Resource Logs

For each selected vault, query Key Vault resource logs using amgmcp_query_resource_log. Keep time range to 1-2 days.

Log Query 1: Failed requests by status code

AzureDiagnostics
| where ResourceType == "VAULTS"
| where TimeGenerated between (datetime(<START>) .. datetime(<END>))
| where httpStatusCode_d >= 400
| summarize count() by httpStatusCode_d, OperationName, bin(TimeGenerated, 1h)
| order by TimeGenerated asc

Log Query 2: Top error operations

AzureDiagnostics
| where ResourceType == "VAULTS"
| where TimeGenerated between (datetime(<START>) .. datetime(<END>))
| where httpStatusCode_d >= 400
| summarize count() by OperationName, httpStatusCode_d, ResultSignature
| order by count_ desc
| take 30

Log Query 3: Throttled requests (429) over time

AzureDiagnostics
| where ResourceType == "VAULTS"
| where TimeGenerated between (datetime(<START>) .. datetime(<END>))
| where httpStatusCode_d == 429
| summarize count() by OperationName, bin(TimeGenerated, 1h)
| order by TimeGenerated asc

Log Query 4: Request volume trend

AzureDiagnostics
| where ResourceType == "VAULTS"
| where TimeGenerated between (datetime(<START>) .. datetime(<END>))
| summarize
    TotalRequests=count(),
    FailedRequests=countif(httpStatusCode_d >= 400),
    ThrottledRequests=countif(httpStatusCode_d == 429),
    AvgDurationMs=round(avg(DurationMs), 2)
    by bin(TimeGenerated, 1h)
| order by TimeGenerated asc

Log Query 5: Authentication failures

AzureDiagnostics
| where ResourceType == "VAULTS"
| where TimeGenerated between (datetime(<START>) .. datetime(<END>))
| where httpStatusCode_d in (401, 403)
| summarize count() by OperationName, CallerIPAddress, identity_claim_appid_g
| order by count_ desc
| take 20

Note: If AzureDiagnostics with ResourceType == "VAULTS" returns no data, diagnostic settings may not be configured. Note it and skip logs for that vault.

---

Classification

Severity	Criteria
CRITICAL	Availability avg < 99.0%, OR SaturationShoebox avg > 75%
WARNING	Availability avg < 99.9%, OR ServiceApiLatency avg > 200ms, OR sustained 429 throttling across multiple time windows, OR sustained 401/403 errors
DORMANT	All metrics return empty timeSeries (no traffic in scan period)
HEALTHY	All metrics within normal ranges

---

Analysis Guidance

For known patterns, deep-dive queries, and correlation techniques, see reference/analysis-patterns.md.

For optional deep-dive queries, see reference/deep-dive-queries.md.

---

Output Format

Present a summary report with these sections:

1. Fleet Inventory

Vault count by region, subscription, and SKU. Flag any vaults not in "Succeeded" provisioning state. Note soft-delete and purge-protection status.

2. Pulse Check Summary

Fleet-wide summary from the keyvault_summary pulse check:

• Total vaults scanned and scan status
• Key fleet-wide metrics (availability, latency, API hits, error counts)
• Breakdown by health status (healthy / warning / critical / dormant)
• Which vaults were selected for deep dive and why each was chosen

3. Deep Dive Findings (Top 7)

For each selected vault:

• Metric timeline: Availability, API latency, API hits, saturation over the scan window
• Anomaly characterization: spike vs sustained, onset time, duration, recovery
• Correlation analysis: which metrics moved together
• Error rate: error results / total API hits percentage

4. Resource Log Findings

For each deep-dived vault:

• Error summary: top error status codes, affected operations, throttling timeline
• Latency analysis: slow operations by type
• Authentication failures: if any, source IPs and application IDs

5. Known Issue Cross-Reference

Compare findings against memory/amg-check-key-vault/report.md. For each known bug, state: still active / improving / worsening / resolved.

6. Action Items

Prioritized list:

• Critical: vaults with availability < 99.0% or saturation > 75%
• High: vaults with sustained throttling (429), high latency, or auth failures
• Medium: vaults with elevated latency, intermittent errors, or approaching saturation
• Low: dormant vaults (investigate if orphaned), informational items

---

Update Known Issues

After presenting findings, update memory/amg-check-key-vault/report.md:

1. Read the current file (create if it doesn't exist).
2. Update status of existing bugs based on today's telemetry.
3. Add new bugs with: severity, vault name, region, metric evidence, log evidence, root cause, recommended action.
4. Update the "Updated" date in the header.

Only add genuine issues: sustained availability drops, persistent throttling, high error rates, or latency degradation.

---

Error Handling

See ${CLAUDE_SKILL_DIR}/reference/error-handling.md for the full recovery table.

---

Reference

• Key Vault resource type: Microsoft.KeyVault/vaults
• ARM ID template: /subscriptions/{SUB}/resourceGroups/{RG}/providers/Microsoft.KeyVault/vaults/{name}
• Resource log table: AzureDiagnostics (with ResourceType == "VAULTS")
• Key status codes: 429 (throttled), 401 (unauthorized), 403 (forbidden), 404 (not found)
• Key metrics: Availability, ServiceApiHit, ServiceApiLatency, ServiceApiResult, SaturationShoebox
• Known issues: memory/amg-check-key-vault/report.md
• Configuration: memory/amg-check-key-vault/config.md

---

First-Run Setup

Run only when Config shows NOT_CONFIGURED. After completing, return to the Workflow above.

1. Discover Datasource UID: Call amgmcp_datasource_list. Filter type == "grafana-azure-monitor-datasource". Prefer uid == "azure-monitor-oob" if multiple match. Abort if zero match.

2. Discover Subscription ID(s): Run this Resource Graph query to list all subscriptions with key vaults, then present the results as a table and ask the user which subscription(s) to use:

resources
| where type == 'microsoft.keyvault/vaults'
| join kind=inner (
    resourcecontainers
    | where type == 'microsoft.resources/subscriptions'
    | project subscriptionId, subscriptionName=name
) on subscriptionId
| summarize KeyVaults=count() by subscriptionId, subscriptionName
| order by KeyVaults desc

Present the results as a table with columns: Subscription Name, Subscription ID, Key Vaults. Then ask the user: "Which subscription ID(s) should I configure for this health check?"

3. Write config: Write memory/amg-check-key-vault/config.md:

# amg-check-key-vault Configuration

User-specific values for the Key Vault health check skill.
This file is auto-generated on first run and can be edited manually.

## Azure Monitor Datasource
- **UID**: {discovered_uid}
- **Name**: {discovered_name}

## Subscriptions
- {subscription_id_1}
- {subscription_id_2}

4. Confirm: Show the resolved config and ask for confirmation before proceeding.