Install
openclaw skills install @huaweicloudskill/huawei-cloud-agentarts-obs-diagnosisObservability no-data diagnosis skill for Huawei Cloud AgentArts agent-ops platform. Systematically diagnoses missing metrics, logs, and traces/sessions in agent-ops by checking subscription metadata, data pipeline (APM/AOM/LTS), query parameters, and authentication. Generates root-cause analysis and fix recommendations. Triggers include: "AgentArts no data", "metrics no data", "logs no data", "traces no data", "session no data", "obs diagnosis", "可观测诊断", "无数据诊断", "查不到指标", "查不到日志", "调用链无数据", "agent-ops 无数据"
openclaw skills install @huaweicloudskill/huawei-cloud-agentarts-obs-diagnosisThis skill systematically diagnoses missing metrics, logs, and traces/sessions in Huawei Cloud AgentArts agent-ops platform. It follows a three-phase diagnostic flow: subscription metadata check → data pipeline troubleshooting → query parameter validation, covering three data sources: APM (traces/metrics), AOM (metrics), and LTS (logs).
Application (OTel SDK/Collector)
│
├── Traces: OTEL SDK → APM (OTLP) → deliverConfig → Kafka → agent-ops
├── Metrics: Application → AOM (Prometheus) → agent-ops
└── Logs: Application → LTS → agent-ops (requires logGroup/logStream metadata)
| Data Type | Source | Required Metadata |
|---|---|---|
| Traces/Sessions | APM | apmBusiness, apmToken |
| Metrics | AOM | promInstance, aomAccessCode |
| Logs | LTS | logGroup, logStream |
export HUAWEI_ACCESS_KEY="<your-access-key-id>"
read -rs HUAWEI_SECRET_KEY; export HUAWEI_SECRET_KEY
export HUAWEI_REGION="cn-north-4"
Install: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html
See references/iam-policies.md for least-privilege policy.
Use AskUserQuestion to determine which data type is missing:
| Option | Data Type | Data Source |
|---|---|---|
| A | Metrics (metrics) | AOM / APM |
| B | Logs (logs) | LTS |
| C | Traces/Sessions (traces/session) | APM |
| D | All missing | All sources |
Record symptoms: complete vs partial absence, error codes, onset time.
| Parameter | Default | Description |
|---|---|---|
region | cn-north-4 | Query region (must match ingestion region) |
domain_id | — | IAM domain ID |
project_id | — | Project ID for LTS queries |
start_time / end_time | — | Millisecond timestamps (13 digits, NOT 10-digit seconds) |
Missing subscription metadata is the most common root cause of no-data issues.
Check APM business and token:
hcloud APM ListBusiness --cli-region=cn-north-4
If no business exists → APM not subscribed. Create business:
hcloud APM CreateBusiness --cli-region=cn-north-4 \
--name="obs-diagnosis-check" \
--display_name="obs-diagnosis-check" \
--descp="diagnosis" \
--cmdb_datasource_type=OTEL
Verify business token:
hcloud APM ShowToken --cli-region=cn-north-4 \
--x-business-id={business_id}
Check AOM Prometheus instance and access code:
hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
hcloud AOM ListAccessCode --cli-region=cn-north-4
If no PromInstance → AOM not configured. Create Prometheus instance
(--prom_type must be one of ECS/CCE/REMOTE_WRITE/CLOUD_SERVICE/ACROSS_ACCOUNT;
--region header is required and must match --cli-region):
hcloud AOM CreatePromInstance --cli-region=cn-north-4 \
--prom_name="obs-diagnosis" \
--prom_type=REMOTE_WRITE \
--region=cn-north-4
Check LTS log groups and streams:
hcloud LTS ListLogGroups --cli-region=cn-north-4
hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}
If no logGroup/logStream → LTS not subscribed for agent-ops.
Check pipeline bottom-up:
hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
hcloud AOM ListAccessCode --cli-region=cn-north-4
hcloud AOM ListMetadataAomPromGet --cli-region=cn-north-4
hcloud AOM ListInstantQueryAomPromGet --cli-region=cn-north-4 \
--query="up" \
--prom_instance_id={prom_id}
resource_tokens, model_tokens,
session_tokens, trace_tokens), filter labels match (gen_ai_resource_type,
resource_id, model_id).hcloud LTS ListLogGroups --cli-region=cn-north-4
hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}
hcloud LTS ListLogs --cli-region=cn-north-4 \
--log_group_id={group_id} \
--log_stream_id={stream_id} \
--start_time=1720000000000 \
--end_time=1720600000000
agentRunId, agentId, traceId, logType,
keyword syntax (LTS query language).LTS.2446 → Unsupported query syntax — fix keyword syntax403 → Permission denied — check agency token (ops_admin_trust)hcloud APM SearchAgent --cli-region=cn-north-4 --x-business-id={business_id}
deliver_otel_trace=true in deliverConfig?hcloud APM SearchTransaction --cli-region=cn-north-4 \
--x-business-id={business_id} \
--start_time=1720000000000 \
--end_time=1720600000000
curl -X POST ${AGENT_OPS_ENDPOINT}/v1/ops/observation/traces \
-H "Authorization: Bearer ${IAM_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"start_time": 1720000000000, "end_time": 1720600000000}'
${IAM_TOKEN} 需在运行期由 IAM token / AppCode 获取后替换(见上 Step 4 agent-ops 查询命令);未替换则为占位符,照抄请求必返回 401。hcloud IAM ListPoliciesV5 --cli-region=cn-north-4
See references/diagnostic-report-template.md for the report format.
| Command | Purpose | Mode |
|---|---|---|
hcloud APM ListBusiness --cli-region=cn-north-4 | List APM businesses | CLI |
hcloud APM ShowBusinessDetail --cli-region=cn-north-4 | Show business details | CLI |
hcloud APM ShowToken --cli-region=cn-north-4 | Get business token | CLI |
hcloud APM ShowAccessPoint --cli-region=cn-north-4 | Get access point | CLI |
hcloud APM SearchAgent --cli-region=cn-north-4 | Search agents | CLI |
hcloud APM SearchTransaction --cli-region=cn-north-4 | Search transactions (traces) | CLI |
hcloud APM ShowSpanSearch --cli-region=cn-north-4 | Search spans | CLI |
hcloud APM ListOpenRegion --cli-region=cn-north-4 | List open regions | CLI |
hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0 | List Prometheus instances | CLI |
hcloud AOM ListAccessCode --cli-region=cn-north-4 | List access codes | CLI |
hcloud AOM ListInstantQueryAomPromGet --cli-region=cn-north-4 | Instant Prometheus query | CLI |
hcloud AOM ListRangeQueryAomPromGet --cli-region=cn-north-4 | Range Prometheus query | CLI |
hcloud AOM ListMetadataAomPromGet --cli-region=cn-north-4 | List Prometheus metadata | CLI |
hcloud LTS ListLogGroups --cli-region=cn-north-4 | List log groups | CLI |
hcloud LTS ListLogStreams --cli-region=cn-north-4 | List log streams | CLI |
hcloud LTS ListLogs --cli-region=cn-north-4 | List logs | CLI |
hcloud IAM ListPoliciesV5 --cli-region=cn-north-4 | List IAM policies | CLI |
POST {AGENT_OPS_ENDPOINT}/v1/ops/observation/traces | Query traces in agent-ops | API |
POST {AGENT_OPS_ENDPOINT}/v1/ops/observation/metrics/trend | Query metric trends | API |
| Parameter | Required | Description | Example |
|---|---|---|---|
region | Yes | Query region | cn-north-4 |
business_id | Yes (traces) | APM business ID | From ListBusiness |
prom_instance_id | Yes (metrics) | AOM Prometheus instance ID | From ListPromInstance |
log_group_id | Yes (logs) | LTS log group ID | From ListLogGroups |
log_stream_id | Yes (logs) | LTS log stream ID | From ListLogStreams |
start_time | Yes | Start time (ms timestamp) | 1720000000000 |
end_time | Yes | End time (ms timestamp) | 1720600000000 |
metric_name | No (metrics) | Metric name to query | resource_tokens |
keywords | No (logs) | LTS keyword search | error AND timeout |
trace_id | No (traces) | Trace ID filter | — |
hcloud <Service> <Operation> --cli-region=<region> [--key=value ...]
| Feature | Description | Example |
|---|---|---|
| Service name | KooCLI Service name (uppercase) | APM, AOM, LTS, IAM |
| Operation name | PascalCase | ListBusiness, ShowToken |
| Region parameter | --cli-region=<value> | --cli-region=cn-north-4 |
| Simple parameter | --key=value | --business_id=xxx |
references/iam-policies.md — Least-privilege IAM policiesreferences/cli-installation-guide.md — hcloud CLI installation guidereferences/verification-method.md — Verification methods and API statusreferences/dataflow-diagram.md — Mermaid data flow and diagnostic flowreferences/acceptance-criteria.md — Acceptance criteria for diagnosisreferences/diagnostic-report-template.md — Diagnostic report template| Scenario | Handling |
|---|---|
| All data types missing | Check subscription status first, then network connectivity |
| Partial data missing (some apps) | Check agent deployment, service.name mapping |
| LTS.2446 error | Fix keyword syntax (escaping, quotes, operators) |
| 403 error (non-claw) | Check IAM tenant permissions and subscription status |
| 403 error (claw) | Check agency (ops_admin_trust) and STS configuration |
| RESOURCE_NOT_EXIST | Resource records missing — confirm CreateClawTenantRelation |
| Metadata missing | Subscribe first (SubscribeOpsObservation) |
| Timestamp 10 digits | Convert to 13-digit milliseconds |
| Region mismatch | Use same region for ingestion and query |
references/iam-policies.md for required permissions.