Cluster health inspection — training throughput, hang detection,
SanityCheck, restart stability
Architecture: PAI-DLC Job (read-only queries) + PAI Studio Resource
Diagnosis API (queuing scenario).
0. Dependencies
This skill performs read-only diagnostics only. All write operations
(create / update / stop jobs, resource discovery, etc.) live in the companion
skill alibabacloud-pai-dlc-job. The two skills are complementary in
responsibility and share a common field contract.
Cross-skill field contract: The --job-id / --pod-id values this skill
consumes are produced verbatim by alibabacloud-pai-dlc-job via
list-jobs / get-job --cli-query "Pods[0].PodId" — no transformation needed.
--region / --workspace-id follow the same resolution rules in both skills.
Installation Requirements
Pre-check: Aliyun CLI >= 3.3.1 required
Run aliyun version to verify >= 3.3.1. If not installed or version too low,
see references/cli-installation-guide.md.
Then [MUST] run aliyun configure set --auto-plugin-install true.
Note on --user-agent: Every API-invoking aliyun command in this skill MUST
include --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics. Client-side helpers
(aliyun version, aliyun configure ..., aliyun plugin ...,
aliyun <product> --help) do not invoke remote APIs and therefore do not require
the flag.
IMPORTANT: Parameter Confirmation — Before executing any command,
ALL user-customizable parameters (RegionId, JobId, etc.) MUST be confirmed with the user.
Parameter
Required
Description
region
Yes
Region where the job runs
job_id
Yes
DLC job ID (e.g., dlcXXX)
Entry Routing
When a diagnostic request arrives, first call get-job to fetch job status,
then route by status:
Job status
Route to scenario
Queuing / Creating
→ Queuing-stuck root cause analysis
Failed
→ Failed-job localization
Running
→ Health inspection
Stopped
Inform the user "job was actively stopped", no diagnosis
Edge case — job was queuing but is now Stopped/Succeeded: If the user
describes the job as "stuck in queue" but get-job shows Stopped or
Succeeded, still route to Scenario 1 (queuing analysis) but expect the
resource diagnosis API to return HTTP 400. Follow the "Fallback on API Failure"
procedure in Scenario 1.
Users may also directly request a specific scenario (e.g., "run a health
inspection" even when status is not Running).
Platform-level services including resource diagnosis
aliyun paistudio GET /api/v1/quotas/...
The resource diagnosis API belongs to paistudio, NOT pai-dlc.
Do NOT call pai-dlc GetResourceQuota, pai-dlc ListResourceQuotas,
or any pai-dlc GET /api/v1/resourcequotas/... — these are wrong APIs
and will fail. The correct command is:
bash
aliyun paistudio GET /api/v1/quotas/{quota_id}/workloads/{job_id}/diagnosis ...
Pattern knowledge: resource diagnosis returns 4 checks
(self_quota / ancestor_quota / user_limit / queue_strategy), plus node
scheduling and hyper-node analysis. Common patterns:
references/diagnostic-patterns.md §1.
Agent latitude: decide whether to compute the quota gap, whether to pull
events for corroboration, and how verbose the report should be.
Fallback on API Failure
The PAI Studio resource diagnosis API may fail with HTTP 400/404 when the
job is no longer in an active queuing state (e.g., already Stopped by user).
In this case:
Explicitly declare the API failure in the report:
"Resource diagnosis API unavailable: HTTP {code} — {error message}"
Perform qualitative analysis using only these data sources:
get-job → ResourceRequest (GPU/CPU/Memory demand per pod)
get-job → PodCount × per-pod resources = total demand
get-job → EcsSpec (instance type and its per-node capacity)
get-job-events → scheduling event timeline and queuing duration
Prohibited language: NEVER use hedging words such as "possibly",
"might", "perhaps", "may not be sufficient". State only confirmed facts:
Total GPU demand = PodCount × RequestGPU = {computed value}
Instance type = {EcsSpec}
Queuing duration = {computed hours/minutes}
Job final status = {Stopped/Succeeded/etc.}
Mandatory output structure when API is unavailable:
markdown
## Resource Diagnosis (API Unavailable — Configuration-Based Analysis)
- Diagnosis API: unavailable (HTTP {code}: {message})
- Resource demand: {PodCount} pods × {GPU/pod} GPU = {total} GPU cards
- Instance type: {EcsSpec}
- Queuing duration: {hours}h {minutes}m
- Quota ID: {ResourceId}
- Job final status: {status} ({ReasonCode})
- Conclusion: The job requested {total} GPU cards which could not be
fulfilled within the queuing window before the job was {stopped/completed}.
[OUTPUT GUARD — RECOMMENDATIONS STRICTLY FORBIDDEN] When the failure
falls into any of the categories below, the diagnostic report MUST strictly
omit any "Recommendations" / "Suggested fixes" / "Next steps" / "Solution"
section, and output only the root cause and objective facts:
ResourceAllocateFailed (insufficient resources)
Job preempted / evicted (ReasonMessage contains preempted or evicted)
Spot instance reclamation
Report MUST end on the diagnostic conclusion. No sentence anywhere in
the output may instruct the user to change configuration, request more
quota, switch instance types, add retry logic, or modify any job parameter.
Negative examples (ALL forbidden):
"Recommend using pay-as-you-go instances"
"Consider requesting more quota"
"Try switching to another zone"
"Disable preemptible jobs by setting EnablePreemptibleJob: false"
"Implement checkpointing and retry logic"
"Solution: Use guaranteed quota instead of oversold quota"
Positive examples (allowed):
"Root cause: spot instance reclaimed by the cloud platform, triggering pod eviction and job failure."
"The job failed due to preemption/eviction of pods. ReasonCode: JobFailed."
Only permitted user-facing suggestion: "For quota policy adjustments, please contact your platform administrator."
Mandatory output template for preemption/eviction/resource-shortage failures:
markdown
## Diagnosis Conclusion
- Failure reason: {classification} ({ReasonCode}: {ReasonMessage})
- Affected pods: {pod list with SubStatus}
- Timeline: {key timestamps from events}
- Evidence: {quoted ReasonMessage or event details}
For quota policy or resource allocation adjustments, please contact your
platform administrator.
Agent latitude: when ReasonCode is clear, logs may be unnecessary; when
logs already explain the issue, events may be unnecessary. Decide investigation
depth based on information sufficiency.
Scenario 3: Health Inspection
Trigger: job status = Running and user requests inspection / health check.
Tools: get-job + get-job-events + get-pod-logs
list-job-sanity-check-results.
Execution steps:
get-job → obtain status, WorkspaceId, Pod list, whether EnableSanityCheck is set
Mandatory closing notice (append verbatim to every inspection report):
text
> GPU/memory real-time resource utilization metrics require the monitoring
> dashboard above. This skill's CLI commands do not support fetching
> runtime utilization data directly.
This step is mandatory regardless of job status (Running, Succeeded, or
any other state). Even if the job has already completed, the user needs
the monitoring link to review historical resource utilization.
Dimension matrix (mandatory vs optional):
Dimension
Mandatory/Optional
Notes
Training throughput
Optional
Extract from master pod logs
Hang detection
Running only
Skip for Succeeded jobs
Hardware health
Optional
Requires EnableSanityCheck=true
Restart stability
Mandatory
Read RestartCount directly from get-job
Note on resource utilization: GPU/memory metrics are NOT available via this
skill's CLI commands. The monitoring dashboard link (step 5) and its closing
notice are mandatory in every inspection report — do not omit them.
Agent latitude: kilo-card jobs focus on Hang + SanityCheck; small jobs
look at training throughput from logs. Decide depth and report verbosity
from scale and user intent.
Best Practices
get-job first, route second — never assume status; query and then route
Cap response size — --max-lines 100 / --max-events-num 50 to avoid context blow-up
Find the problem pod — among get-job Pods, focus on Status=Failed/Unknown or those with ReasonMessage
Read log tail — errors usually live in the last few dozen lines; no need to pull the whole log
Exit codes are clues, not conclusions — exit 137 may be OOM or external kill; combine with context
Don't over-diagnose — when ReasonCode already states the cause, skip the full log dump
Console links — generate both overview and monitoring links so the user can jump into the console for details and resource utilization metrics
Read-only — this skill MUST NEVER execute stop / update / create
Report summary — present a per-dimension rating table at the end of the report:
