Manage deep learning training jobs on Alibaba Cloud PAI-DLC (Platform for AI - Deep
Learning Containers) service.
Scenario Description
PAI-DLC is a distributed training service provided by Alibaba Cloud's AI Platform PAI,
supporting:
Job Creation and Execution — Create distributed training jobs for TensorFlow,
PyTorch, XGBoost, and other frameworks
Job Monitoring — Get job status, logs, events, and monitoring metrics
Compute Health Check — Check health status of GPU and other compute devices
Job Management — Update and stop jobs
Architecture: PAI Workspace + DLC Job + Computing Resources (ECS public pay-as-you-go
or Lingjun dedicated quota) + AIWorkSpace catalog (images / datasets / code sources /
quotas / workspaces).
Installation Requirements
Pre-check: Aliyun CLI >= 3.3.1 required
Run aliyun version to verify version >= 3.3.1. If not installed or version is too low,
see references/cli-installation-guide.md for
installation instructions.
Then [Required] run aliyun configure set --auto-plugin-install true to enable
automatic plugin installation.
Note on --user-agent: Every API-invoking aliyun command in this skill MUST
include --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job. Client-side helpers
(aliyun version, aliyun configure ..., aliyun plugin ...,
aliyun <product> --help) do not invoke remote APIs and therefore do not require
the flag.
Network timeout & retry (rule --help doesn't enforce):aliyun CLI
defaults to 10s connect / 10s read with no retry. For long-running flows
(large list, slow region) explicitly raise via global flags
--connect-timeout 15 --read-timeout 30 --retry-count 2. Never rely on the
default for user-confirmed high-risk calls (stop-job / delete-*).
This skill does not require any custom environment variables. Credentials are handled
by the Alibaba Cloud CLI configuration (see Authentication below). Optionally:
Variable
Required
Purpose
ALIBABA_CLOUD_PROFILE
Optional
Selects a non-default aliyun configure profile
ALIBABA_CLOUD_REGION_ID
Optional
Default region when --region is omitted (still recommended to pass --region explicitly)
Do NOT export ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET from
within this session; configure them outside (aliyun configure or shell profile).
Authentication Configuration
Pre-check: Alibaba Cloud Credentials Required
Security Rules:
NEVER read, echo, or print AK/SK values (e.g., echo $ALIBABA_CLOUD_ACCESS_KEY_ID is FORBIDDEN)
NEVER ask the user to input AK/SK directly in the conversation or command line
NEVER use aliyun configure set with literal credential values
ONLY use aliyun configure list to check credential status
bash
aliyun configure list
Check the output for a valid profile (AK, STS, or OAuth identity).
AIWorkSpace authorization note:Image / DataSourceId / CodeSourceId /
WorkspaceId field values for create-job come from the
AIWorkSpace resource-discovery APIs. --resource-id (QuotaId) is manually provided by the user.
RAM users MUST hold the corresponding
AIWorkSpace-namespaced permissions listed above (do not abbreviate as aiworkspace:*).
Parameter Confirmation
Authoritative parameter reference is aliyun pai-dlc <cmd> --help (must-read
before every call). This skill only documents what --help does not tell
you: cross-field rules, cross-product dependencies, hidden behaviors, business
labels, and reject patterns. Whenever a rule below contradicts --help, the
reason is stated inline.
Confirm before call: all user-customizable values (region, names, CIDR,
specs, etc.) MUST be confirmed with the user — never assume defaults.
Hard rules that override --help
Rule
Why this skill overrides --help
--workspace-id is always required
--help marks it optional, but server silently falls back to the user's default workspace if omitted → job often lands in the wrong workspace. Always confirm with user.
--job-specs[].Image MUST be a verbatim ImageUri from aiworkspace list-images
Cross-product contract; --help only describes the field type. See §7.6 red line.
--data-sources[].DataSourceId from aiworkspace list-datasets; --code-source.CodeSourceId from list-code-sources
Cross-product discovery; --help cannot point you to the source product.
EcsSpec and ResourceConfig MUST NOT both appear in the same TaskSpec.
Also required before create-job:--job-specs[].Image MUST come from
aliyun aiworkspace list-images; --data-sources[].DataSourceId from
list-datasets; --code-source.CodeSourceId from list-code-sources.
Full discovery flow → see §7.6.
Distributed architecture choices:
Topology
JobSpecs shape
Single-node
One Worker only
TFJob PS-Worker
Both PS (CPU) and Worker (GPU) roles
PyTorch multi-node
One Worker with PodCount > 1
Optional flags: --enable-gang-scheduling true (all-or-nothing scheduling),
Settings.EnableRDMA: true (high-performance network for multi-node GPU),
Settings.EnableSanityCheck: true (GPU health verification).
All commands below require --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job (omitted in snippets for brevity — see Installation Requirements).
Labels rules (not in --help): comma-separated Key=Value pairs, no
JSON / no spaces. Values MUST come from list-image-labels — never invent.
Do not pass --workspace-id to list-images when discovering official
public images (they are global). Pass --workspace-id only when filtering
custom / private images scoped to a specific workspace.
RED LINE:--job-specs[].Image MUST be a verbatim ImageUri (not
Name / ImageId).
7.7 Job Lifecycle Management (Stop / Update / Web Terminal)
Stop is a high-risk operation. Before proceeding, query status with
get-job, present the result to the user, and require explicit confirmation.
Rules --help doesn't tell you (update-job silent-no-op family):
Stop Job applies only when status is Running or Queuing.
update-job --priority takes effect only when (a) the job uses
quota resources (--resource-id) AND (b) status is Creating,
Queuing, or EnvPreparing. Once the job enters Running or later,
priority cannot be modified — the API returns 200 OK but the change
is silently NOT applied. Always pre-check status with get-job.
update-job --accessibility takes effect immediately in any status.
update-job does NOT expose --display-name (--help lists only
--job-id, --accessibility, --description, --job-specs, --priority).
To rename a job, recreate it.
For the full pre-check + confirmation + execution templates, plus the
update-job low-risk path and get-web-terminal / get-token sharing
commands, see references/job-management.md.
7.8 Ecs Spec Discovery
Discover available instance types; the returned EcsSpec value goes
verbatim into --job-specs[].EcsSpec.
list-ecs-specs does not support --sort-by — even values shown as
valid in --help (e.g. CPU / GPU / Memory / GmtCreateTime) are
rejected by the server. Always omit --sort-by here and sort the JSON
output client-side with jq — e.g.
... | jq '.EcsSpecs | sort_by(-.AcceleratorNumber)'.
Success Verification Method
For step-by-step end-to-end verification scripts (resource discovery →
CreateJob → log query → cleanup), see
references/verification-method.md.
Quick verification:
get-job → Status should be Creating / Queuing / Running shortly after
create-job returns.
list-jobs --status Running → Should return the freshly created Job until it
finishes or is stopped.
get-pod-logs → Should return non-empty log content once the Pod is past
EnvPreparing.
Command Tables
The full command index (5 categories × ~40 commands, with plugin
attribution) is consolidated in
references/related-apis.md §1.
Best Practices
Items below are decision rules and operational habits — not parameter
values (those live in --help).
Job naming — use meaningful, sortable names: project-model-date
(e.g. resnet50-imagenet-20260320). Recreate (not update-job) is the
only way to rename.
Resource sizing — pick GPU type / count by model & dataset size. Verify
availability with list-ecs-specs --accelerator-type GPUbefore picking
EcsSpec (see §7.8).
Diagnose early — follow the order get-job → get-job-events →
get-pod-logs → get-pod-events. Cap responses (--max-lines 100,
--max-events-num 50) to keep agent context lean.
Priority adjustment — prefer setting --priority at create-job time.
Post-creation update-job --priority only works for quota jobs in
Creating / Queuing / EnvPreparing phase (§7.7); once Running,
priority cannot be modified.
Cost control — use --job-max-running-time-minutes as an auto-stop guard
for every long-running experiment. Spot via SpotSpec reduces cost at the
risk of preemption.
Health check — enable Settings.EnableSanityCheck: true for GPU
training to catch faulty devices before training starts.
Resource cleanup — stop-job on completed jobs to free quota.
Idempotency on writes — PAI-DLC create-* APIs do NOT expose
--client-token (verified via aliyun pai-dlc create-job --help). Network
retries can therefore create duplicate Jobs. Mitigation: before re-issuing
a failed create-*, run list-jobs --display-name <name> to detect a
half-committed prior attempt.
