huawei-cloud-cloudrobo-dataset
Manage CloudRobo data processing (proc-tasks) and data evaluation (eval-tasks) — query algorithms, create/list/show/update/delete/copy tasks; poll task status until terminal; retrieve system and job logs; download logs; query task events (normal/warning); preview output data and frames; discover available algorithms from the asset marketplace; orchestrate processing→evaluation pipelines; batch-manage tasks across workspaces; diagnose failures via log analysis; and onboard custom data-processing operators (five-phase: deploy mode → code adaptation → image build/SWR push or OBS delivery → asset registration → runtime verification). Chained evaluation only; standalone simulation evaluation belongs to huawei-cloud-cloudrobo-eval. Multi-stage data-cleaning pipelines belong to huawei-cloud-cloudrobo-data-pipeline. Triggers include: data processing task management, data evaluation task management, task copy, task status polling, task events, log retrieval for troubleshooting, output data preview, algorithm discovery, task pipeline orchestration, batch task management, failure diagnosis, custom operator onboarding, dataset, proc-tasks, eval-tasks. Also prepare LeRobot manifest.yaml from a local ROS2 bag through terminal conversation or the bag2manifest Web workbench. Triggers include bag2manifest, ROS2 bag转LeRobot配置, 对话生成manifest.yaml, 启动bag2manifest Web服务, 话题映射.
Windows / PowerShell: Examples use bash syntax. To run on Windows PowerShell:
- Flatten
\line continuations to a single line, or end lines with a backtick.- Set env vars with
$env:NAME="value"instead ofexport NAME="value".- Single-quoted JSON
'{"a":"b"}'works as-is.
Overview
The cloudrobo-dataset skill manages the full lifecycle of CloudRobo data processing
(proc-tasks) and data evaluation (eval-tasks): task CRUD + copy,
status polling, log retrieval (system + job) and download, task event
queries (get-events), output data preview and frame extraction,
marketplace algorithm discovery, processing→evaluation pipeline orchestration, batch
management, failure diagnosis, and custom operator onboarding.
Applicable scenarios: Task management (CRUD + copy), troubleshooting (log retrieval + error analysis), algorithm discovery, result verification (preview + frames), pipeline orchestration, batch management, analytics.
Architecture:
Agent / LLM
│
├── CLI → cloudrobo dataset proc <command> (proc-tasks)
│ cloudrobo dataset eval <command> (eval-tasks)
├── SDK → DatasetClient (Python)
│
▼
cloudrobo-service (REST API)
/v1/data-eng/proc-tasks/*
/v1/data-eng/eval-tasks/*
CloudRobo task operations target the cloudrobo-service backend and require a workspace_id
(default workspace is used unless --workspace-id overrides it). Algorithm and dataset discovery is a
cross-package operation that calls the cloudrobo-asset-manager service via cloudrobo asset.
Prerequisites
See references/cli-installation-guide.md for CLI installation, AK/SK authentication, and
workspace configuration. CloudRobo task operations require a valid workspace_id (default or
explicit --workspace-id).
Local ROS bag manifest preparation
For local bag profiling or LeRobot manifest.yaml preparation, read
Bag to manifest. It covers two modes: terminal conversation
using scripts/bag2manifest/generate_manifest.py, and a local Web workbench using
scripts/bag2manifest/start_web.py. This stage prepares conversion configuration locally;
it does not create a CloudRobo task or require a workspace. Generate ROS Manifest 2.0
according to ROS manifest contract. Follow the user’s
automatic or interactive decision preference; Web uses a shared field editor and sampled previews.
Workflow
Interactive Task Creation
All interactive creation workflows follow the Interaction Rules: at most one tool call per turn, display query results before asking the next question.
Proc-Task Creation (10 steps): Task name → Description → Algorithm type (PRESET_ASSETS/WORKSPACE_ASSETS/OBS_ASSETS) → Algorithm configuration with field mapping → Environment variables (+ operator inputs when declared) → Job type and resources (CCE/K8S or CCE_RAY/RAY, resource pool, instance spec) → Dynamic storage (public pool only) → Input datasets (multi-dataset loop) → Output configuration → Confirm and submit. See references/interactive-workflow.md for detailed steps.
Eval-Task Creation (7 steps): Task name → Description → Resource pool and instance spec → Evaluation algorithm (+ operator inputs when declared) → Dataset (single) → Robot config → Confirm and submit. See references/interactive-workflow.md for detailed steps.
Task Polling and Result Handling
- Poll status —
dataset proc wait-task/dataset eval wait-task(recommended 15s interval), report each state change to user, until terminal state (SUCCEEDED/FAILED/DELETED). Ifwait-tasktimes out while the task is still non-terminal, re-runwait-task— do not create a new task - On completion:
- SUCCEEDED: view processing result via
get-preview, output system log - FAILED: ask user whether to view logs, output system and job logs
- SUCCEEDED: view processing result via
Eval tasks: After SUCCEEDED, extract report path from target_report_path in task detail, use dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> to get OBS temporary link.
Pipeline Workflow (Task Orchestration)
Scenario: data conversion → data evaluation → download artifact
- Run processing task, wait for SUCCEEDED
- Extract
target_pathandtarget_asset_idfrom task detail - For the eval task, set top-level
dataset_type/dataset_id(=target_asset_id)/dataset_name/dataset_path(=target_path) on the task-config - Create evaluation task, wait for SUCCEEDED (
dataset eval wait-task --task-id <eval-task-id>) - Get report:
dataset eval get-preview --file-name <path-from-target_report_path>(removeobs://<bucket>/prefix) - Download artifacts via
download-asset
Batch Workflow (Batch Tasks)
Scenario: run same algorithm across multiple datasets
- Get dataset list (
asset list-assets --type dataset) - Create processing task per dataset (reuse algorithm/resource config from first; only input dataset differs)
- Collect task_ids, poll status
- Summarize success/failure/in-progress counts
- For failed tasks: view logs or re-run by copying the task (
copy-task)
⚠️ Semantics distinction: this Batch Workflow is opt-in — N tasks, one per dataset, only when the user explicitly asks for batch processing. By DEFAULT (including the custom-operator Phase 4 flow), multiple data sources are processed by ONE task via
dataset_configsN entries (operator consumes theSOURCE_DATASET_PATHJSON map). Never default to one-task-per-dataset for a multi-source input.
Diagnosis Workflow (Failure Diagnosis)
Scenario: task FAILED → analyze logs, suggest fixes
- Get system and user logs
- Analyze error patterns: OOM → increase
worker_spec.memory; path not found → checkdataset_configs; image pull failure → checkimage; parameter error → checkalgo_entrance/envs; access denied → checkcatalog_id/permissions - Output diagnosis and fix suggestions
- After confirmation, modify via
update-task(onlyname/description) or re-run viacopy-task
Custom operator failures → fetch job-std-output.log and apply the P0-P3 patterns in references/custom-operator/custom-operator-workflow.md §Failure Analysis Guide.
Analytics Workflow (Data Analysis)
Aggregate task stats by status, algorithm ranking, time trend, failure rate. Output analysis report with root-cause suggestions.
Long-Running Task Workflow
Start wait-task with timeout. On timeout, report status and elapsed time; user decides: continue / view logs / terminate.
CLI Command Format Standard
cloudrobo dataset proc <command> [OPTIONS]
cloudrobo dataset eval <command> [OPTIONS]
| Feature | Description | Example |
|---|---|---|
| Command group | dataset proc (proc-tasks), dataset eval (eval-tasks) | cloudrobo dataset proc |
| Subcommand | kebab-case | create-task, list-tasks, show-task |
| Workspace override | --workspace-id <id> | --workspace-id abc-123 |
| Output format | JSON to stdout | out(result) |
| Dry-run | --dry-run (create/copy only) | Preview without executing |
| Boolean flag | --is-system true / --all / --is-download | --is-system true |
| Event query | --job-name <name> | cloudrobo dataset proc get-events --task-id <id> --job-name <name> |
Core Commands
Task Management (proc-tasks)
Create a data processing task
cloudrobo dataset proc create-task --name <task-name> --algo-type PRESET_ASSETS --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800] [--dry-run]
- SDK:
client.create_task(req, workspace_id=None) - API:
POST /v1/data-eng/proc-tasks
Required task_config fields: name, algo_type (PRESET_ASSETS / WORKSPACE_ASSETS / OBS_ASSETS, based on Step 3 selection), algo_name, algo_entrance,
image, algo_id (required unless algo_type=OBS_ASSETS), catalog_id (workspace's), resource_pool_type, cluster_type,
task_framework_type, dataset_configs (JSON string array, multi-dataset input),
output_type, output_path (required when output_type=UDF_OBS_ASSET), output_name, head_spec & worker_spec (must include cpu/memory), worker_num,
evs_spec. For algo_type=OBS_ASSETS, algo_path and job_local_path replace algo_id.
Optional: description, envs, inputs (operator input mounts, see
Operator Inputs), resource_id, dedicated_pool_name.
See references/task-config-catalog.md for full field mapping.
algo_typesemantics (algorithm reference way — NOT the deploy mode):
WORKSPACE_ASSETS— references a workspace-registered algorithm asset viaalgo_id(required, asset linkage) and requiresalgo_path(= assetcode_dir, mount source) +job_local_path(mount path) — verified: omitting them yields//entrypoint.pyruntime failure. Applies to Mode A (image_source=custom) and Mode B 空间资产-算法 (image_source=custom, algorithm asset links the image asset) — NOT Mode-A-only.PRESET_ASSETS— references a platform preset algorithm viaalgo_id(required).OBS_ASSETS— mounts the customer OBS code directly (algo_path+job_local_pathrequired;algo_idoptional). This is Mode B OBS算法. Seereferences/task-config-catalog.mdfor the algo_type-specific field matrix.
⚠️ Non-Empty Validation: Except description, all parameters cannot be empty values (empty string "", None, empty dict {}). The SDK validates this before API submission. head_spec/worker_spec must include cpu and memory (add gpu/npu as 0 when unused). dataset_configs must contain at least 1 dataset entry; empty array "[]" is invalid.
List tasks
cloudrobo dataset proc list-tasks [--status RUNNING|SUCCEEDED|FAILED] [--name <name>] [--order-by start_at|update_at|finish_at|create_at] [--order DESC|ASC] [--offset <n>] [--limit <n>] [--user-id <id>] [--algo-name <name>] [--output-name <name>] [--workspace-id <id>]
- SDK:
client.list_tasks(workspace_id=None, statuses=..., name=..., order_by=..., order=..., offset=..., limit=..., user_id=..., algo_name=..., output_name=...) - API:
GET /v1/data-eng/proc-tasks?workspace_id=<id>&statuses=<status>
Show task detail
cloudrobo dataset proc show-task --task-id <task-id>
- SDK:
client.get_task_detail(task_id) - API:
GET /v1/data-eng/proc-tasks/{task_id}
Update a task
cloudrobo dataset proc update-task --task-id <task-id> --task-config '{"name":"<new-name>"}'
Only name and description can be modified.
- SDK:
client.update_task(task_id, task_config: dict) - API:
PATCH /v1/data-eng/proc-tasks/{task_id}
Delete a task
cloudrobo dataset proc delete-task --task-id <task-id>
- SDK:
client.delete_tasks([task_id_1, task_id_2])(accepts a list; CLI deletes one at a time) - API:
DELETE /v1/data-eng/proc-tasks?ids=id1,id2
Copy a task
cloudrobo dataset proc copy-task --task-id <task-id> --name <new-name> --output-name <new-output-name> [--task-config '<json>'] [--wait] [--timeout 1800] [--dry-run]
--name and --output-name are required. Other fields are fetched from show-task
(get_task_detail) and can be overridden via --task-config. The new task starts in PENDING.
Keep every input's exact dataset_configs[].version_id; never replace it with the asset's
latest version. Check the copied task detail retains the original asset/version/path.
- SDK:
client.copy_task(task_id, req={"name": ..., "output_name": ...}) - API:
POST /v1/data-eng/proc-tasks/{task_id}/copy
Note: proc-tasks do not support restart — to re-run a task, use copy-task.
Task Monitoring
Wait for task to reach terminal state
cloudrobo dataset proc wait-task --task-id <task-id> [--timeout 1800] [--interval 10]
Terminal states: SUCCEEDED, FAILED, DELETED
- SDK:
client.wait_task(task_id, timeout=1800, interval=10, on_status=callback) - API:
GET /v1/data-eng/proc-tasks/{task_id}(polled)
Retrieve task logs
# Step 1: List log files to get file_path
cloudrobo dataset proc get-log --task-id <id> --is-system true
cloudrobo dataset proc get-log --task-id <id> --is-system false
# Step 2: Get log content (default: latest 64KB tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path>
# Get full log (not just tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path> --all
- SDK:
client.list_log_files(task_id, is_system=True)→client.get_task_log(task_id, file_name, file_path, start_byte, end_byte)orclient.get_task_log_tail(task_id, file_name) - API:
GET /v1/data-eng/proc-tasks/{task_id}/logs?is_system=true|falseGET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}?start_byte=&end_byte=&file_path=&job_id=
Download a log file
cloudrobo dataset proc download-log --task-id <id> --file-name <name> --file-path <path>
- SDK:
client.download_task_log(task_id, file_name, file_path) - API:
GET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}/download
Get task resource usage
cloudrobo dataset proc get-resource-usage --task-id <task-id> --metric CPU_UTIL|CPU_USED_CORE|MEM_UTIL|MEM_USED_MB|NETWORK_TX_RATE|NETWORK_RX_RATE|DISK_READ_KB|DISK_WRITE_KB --start <unix-ts-sec> --end <unix-ts-sec> --step <10-3600>
- SDK:
client.get_task_resource_usage(task_id, metric, start, end, step) - API:
GET /v1/data-eng/proc-tasks/{task_id}/resource-usage?metric=&start=&end=&step=
Get task events
cloudrobo dataset proc get-events --task-id <task-id> --job-name <job-name>
Returns the task's runtime events (NORMAL/WARNING), first/last occurrence time (Unix timestamp), description, and count.
- SDK:
client.get_task_events(task_id, job_name) - API:
GET /v1/data-eng/proc-tasks/{task_id}/events?job_name=<job-name>
Data Preview
Preview task output data
Get OBS temporary download link for a dataset file. file_name = OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../data/chunk-000/file-000.parquet).
cloudrobo dataset proc get-preview --task-id <task-id> --file-name <obs-path-without-bucket> [--is-download]
- SDK:
client.get_task_preview(task_id, file_name, is_download=False) - API:
GET /v1/data-eng/proc-tasks/{task_id}/preview?file_name=<file_name>&is_download=<bool>
Get task frames
Query directory file list of task input/output datasets. prefix = dataset OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../). Pagination uses --max-keys (≤50) and --next_marker (value from the previous response).
cloudrobo dataset proc get-frames --task-id <task-id> --prefix <obs-path-without-bucket> [--max-keys <1-50>] [--next_marker <marker>]
- SDK:
client.get_task_frames(task_id, prefix, max_keys=..., next_marker=...) - API:
GET /v1/data-eng/proc-tasks/{task_id}/frames?prefix=<prefix>&max_keys=&next_marker=
Evaluation Tasks (eval-tasks)
Evaluation tasks differ from processing tasks in field names, deletion granularity, and supported
operations. See the full comparison table in references/task-config-catalog.md.
Create an evaluation task
cloudrobo dataset eval create-task --name <task-name> --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800]
- SDK:
client.create_eval_task(req, workspace_id=None) - API:
POST /v1/data-eng/eval-tasks
task-config field description:
The eval-task task-config uses top-level single-value dataset fields (dataset_type, dataset_id, dataset_name, dataset_path) plus a single-item dataset_configs JSON-array string. For BUILD_IN_ASSET, include the exact input version_id in that entry and keep asset/name/path/type consistent with the top-level fields. See version handoff and cloning.
Common required fields: name, algo_id, algo_name, algo_entrance, image, dataset_type, dataset_name, dataset_path, dataset_configs, resource_pool_type, worker_spec. dataset_id is additionally required when dataset_type=BUILD_IN_ASSET.
Optional: description, resource_id, dedicated_pool_name, workspace_id, envs, inputs
(operator input mounts, see Operator Inputs).
⚠️ Key reminders:
- Unlike proc-tasks, eval task-config does not include
algo_type,catalog_id,cluster_type,task_framework_type,head_spec,worker_num,evs_spec, oroutput_type/output_path/output_name worker_specmust includecpuandmemorykeysrobot_confighas been removed from eval tasks — never send it. Config files such as the robot description (robots.yaml / URDF) are mounted from OBS only through the operator's declaredinputs(e.g.ROBOT_CONFIG_PATH). Without a mount, run with the operator's built-in/default configuration, including manifest-mode input; explain unavailable robot-dependent metrics. Do not ask for a path by default.- If the CLI/SDK still rejects an eval task for missing
robot_config, the client is outdated: upgrade it; never add the field, invent a path or bypass the client. See config mount guidance. dataset_namecannot be empty (extract from OBS path last directory for UDF_OBS_ASSET)dataset_typevalues:BUILD_IN_ASSET/UDF_OBS_ASSETalgo_typeis not a field; the algorithm is identified byalgo_id/algo_name/algo_entrance/image
See references/task-config-catalog.md for full field mapping and eval-task field differences.
List evaluation tasks
cloudrobo dataset eval list-tasks [--status <status>] [--name <name>] [--order-by start_at|update_at|finish_at|create_at] [--order DESC|ASC] [--offset <n>] [--limit <n>] [--user-name <name>] [--dataset-name <name>] [--workspace-id <id>]
- SDK:
client.list_eval_tasks(workspace_id=None, statuses=..., name=..., order_by=..., order=..., offset=..., limit=..., user_name=..., dataset_name=...)(--limitdefaults to 20) - API:
GET /v1/data-eng/eval-tasks?workspace_id=<id>
Show evaluation task detail
cloudrobo dataset eval show-task --task-id <task-id>
- SDK:
client.get_eval_task_detail(task_id) - API:
GET /v1/data-eng/eval-tasks/{task_id}
Update an evaluation task
cloudrobo dataset eval update-task --task-id <task-id> --task-config '{"name":"<new-name>"}'
Only name and description can be modified.
- SDK:
client.update_eval_task(task_id, task_config: dict) - API:
PATCH /v1/data-eng/eval-tasks/{task_id}
Delete an evaluation task (single granularity)
cloudrobo dataset eval delete-task --task-id <task-id>
- SDK:
client.delete_eval_task(task_id) - API:
DELETE /v1/data-eng/eval-tasks/{task_id}
Note: eval-tasks deletion is single-task granularity (delete-task, not batch). eval-tasks do
not support restart — to re-run, use copy-task.
Copy an evaluation task
cloudrobo dataset eval copy-task --task-id <task-id> --name <new-name> [--task-config '<json>'] [--wait] [--timeout 1800] [--dry-run]
Only --name is required (--output-name does not apply to eval tasks). Other fields are fetched
from eval show-task (get_eval_task_detail) and can be overridden via --task-config. The new
task starts in PENDING.
Keep both top-level dataset identity and single-item dataset_configs with the original
exact version. For a historical task without version metadata, follow the
version handoff rules before copying.
- SDK:
client.copy_eval_task(task_id, req={"name": ...}) - API:
POST /v1/data-eng/eval-tasks/{task_id}/copy
Wait for evaluation task to reach terminal state
cloudrobo dataset eval wait-task --task-id <task-id> [--timeout 1800] [--interval 10]
Terminal states: SUCCEEDED, FAILED, DELETED. Status changes are printed as they occur.
On timeout the command returns the latest detail while the task may still be running —
re-run wait-task instead of creating a new task. To wait for several evaluation tasks in
parallel, create them without --wait, record each task_id, then run wait-task per task.
- SDK:
client.wait_eval_task(task_id, timeout=1800, interval=10, on_status=callback) - API:
GET /v1/data-eng/eval-tasks/{task_id}(polled)
Retrieve evaluation task logs
# Step 1: List log files
cloudrobo dataset eval get-log --task-id <id> --is-system true
cloudrobo dataset eval get-log --task-id <id> --is-system false
# Step 2: Get log content (default: latest 64KB tail; --all for full content)
cloudrobo dataset eval get-log --task-id <id> --file-name <name> --file-path <path> [--all]
- SDK:
client.list_eval_log_files(task_id, is_system=True)+client.get_eval_task_log(task_id, file_name, ...)/client.get_eval_task_log_tail(...) - API:
GET /v1/data-eng/eval-tasks/{task_id}/logs?is_system=true|falseGET /v1/data-eng/eval-tasks/{task_id}/logs/{file_name}?file_path=&job_id=
Get evaluation task preview
file_name is the OBS path of the report file (excluding bucket name). Extract from task detail's target_report_path by removing obs://<bucket-name>/ prefix.
For example: if target_report_path is obs://cloudrobo-test-203/eval_task/report/abc-123/diversity_evaluation_report.pdf, then file_name = eval_task/report/abc-123/diversity_evaluation_report.pdf.
cloudrobo dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> [--is-download]
- SDK:
client.get_eval_task_preview(task_id, file_name, is_download=False) - API:
GET /v1/data-eng/eval-tasks/{task_id}/preview?file_name=<obs-path-without-bucket>&isDownload=
Returns OBS temporary URL. Use --is-download for download link, omit for preview link. Save or download promptly.
Download an evaluation log file
cloudrobo dataset eval download-log --task-id <id> --file-name <name> --file-path <path> [--is-download]
- SDK:
client.download_eval_task_log(task_id, file_name, file_path) - API:
GET /v1/data-eng/eval-tasks/{task_id}/logs/{file_name}/download
Get evaluation task events
cloudrobo dataset eval get-events --task-id <task-id> --job-name <job-name>
Returns the eval task's runtime events (NORMAL/WARNING), first/last occurrence time (Unix timestamp), description, and count.
- SDK:
client.get_eval_task_events(task_id, job_name) - API:
GET /v1/data-eng/eval-tasks/{task_id}/events?job_name=<job-name>
Note: eval-tasks do not support get-frames or get-resource-usage (proc-tasks only).
Algorithm Discovery
List available algorithms
cloudrobo asset list-publication-assets --type algorithm --tags "Data Processing" [--name <fuzzy-name>] [--limit 20]
- SDK (cross-package):
asset_client.list_publication_assets(type="algorithm", sub_type="data_processing", limit=20) - API: Cross-package — calls the asset service, not the dataset service directly.
Each algorithm includes ext_metadata with engine.image_url, command, and
environment_variables needed for task creation. Use --tags "Data Evaluation" to discover
evaluation algorithms.
Query fresh every time: re-run
list-publication-assetsbefore every task submission and extractimage/command/inputsfrom the current result. Never reuse operator info from a previous run — the platform updates algorithm assets (new image versions, fixed commands), and a stale image can silently fail (e.g. manifest mounts unsupported in the old image).
Operator Inputs (ext_metadata.inputs)
ext_metadata.inputs declares optional/required external inputs (config/model paths)
that the algorithm mounts at task run time. Read ext_metadata.inputs from the current
query when constructing task_config (proc or eval) and carry the inputs to be mounted in the
task_config inputs field:
inputsis a JSON-array string field (likedataset_configs/envs)- Each provided element is the complete
ext_metadata.inputs[]element, copied as returned by the current query (do not add or strip fields), withurl_pathset to the user-selectedobs://path. Never useobs_pathas an inputs key. required: trueinputs with a built-inurl_pathneed no user action and do not need to be listed ininputs;required: trueinputs with an emptyurl_pathmust be supplied by the user and included ininputsrequired: falseinputs are optional — do not prompt by default; omit them (omit the wholeinputsfield when nothing needs to be mounted)- If the operator's current
ext_metadatahas noinputs, omit theinputsfield
"inputs": "[{\"name\":\"<name as returned>\",\"label\":\"<as returned>\",\"required\":false,\"url_path\":\"obs://bucket/config/\",\"input_type\":\"<as returned>\",\"description\":\"<as returned>\",\"access_method\":\"env\"}]"
See references/task-config-catalog.md → "Operator Inputs Mount" for the full structure,
known operator inputs (reference only), and priority rule.
Input datasets do not universally require YAML files. Use the selected operator's built-in
robot environment/default configuration, existing dataset config, or optional external
config mount as appropriate. Only when a YAML config file is actually needed, apply the
manifest candidate rules;
manifest mode alone does not require an external file or an evaluation config mount.
Custom operator onboarding
For custom data processing operators (user code that needs containerizing, SWR push, or OBS delivery + asset registration), follow the five-phase pipeline in Custom Operator Onboarding below.
Custom Operator Onboarding 自定义算子入驻
Onboard user-developed data-processing operators to CloudRobo in an orchestrated
five-phase pipeline. Core design principle: never modify the user's original code —
the skill copies the user's code read-only into a temporary work copy
(/tmp/custom-operator-work-{timestamp}/) and generates all adaptation artifacts
(entrypoint.py, Dockerfile, operator_config.json) there only.
Phase 0 Deploy Mode + Assets Phase 1 Code Adaptation Phase 2 Build & Push Phase 3 Asset Registration Phase 4 Runtime Verification Phase 5 Cleanup & Report
├─▶ A = image-inbuilt(custom) work copy + entrypoint.py docker build + push (new image + algorithm assets WORKSPACE_ASSETS + algo_id Part 1A server junk (always)
└─▶ B = OBS-mounted(preset) code → OBS (deliver) SKIP (no build) 空间资产-算法: preset image + algo /log/frames verify Part 1B platform junk (ask) + report
OBS算法: SKIP (no assets) 空间资产-算法: WORKSPACE_ASSETS+algo_id
OBS算法: OBS_ASSETS (no algo_id)
failure → Failure Analysis Engine (job-std-output.log → pattern → fix) → after final result, cleanup is MANDATORY
Three mandatory customer choices (never inferred; re-ask until unambiguous — a wrong guess
mis-routes Phase 3 asset registration and the task's algo_type, so each choice must be asked):
- Deploy mode (asked FIRST) — Mode A image-inbuilt (
custom): every submission rebuilds- pushes a new image (timestamp tag; SWR login auto-generated from AK/SK — on
auto-credential generation/login failure, classify the cause (AK/SK invalid/expired, SWR
not enabled / IAM permission missing, region mismatch, network) and ask the user to fix /
provide a full
docker logincommand, seereferences/custom-operator/custom-operator-workflow.md§Step 2.0); Mode B OBS-mounted (preset): code delivered to OBS, no build/push, preset image registered as an asset (空间资产-算法 only; the algorithm asset links it viaengine.asset_id+engine.asset_version_id).
- pushes a new image (timestamp tag; SWR login auto-generated from AK/SK — on
auto-credential generation/login failure, classify the cause (AK/SK invalid/expired, SWR
not enabled / IAM permission missing, region mismatch, network) and ask the user to fix /
provide a full
- Mode B task-creation way (asked when Mode B is chosen) — 空间资产-算法:
WORKSPACE_ASSETS+algo_id(Phase 3 asset) +algo_path(= assetcode_dir) +job_local_path+ absolutealgo_entrance(asset linkage, registration runs); Phase 3 算法资产 engine 必须通过asset_id+asset_version_id关联创建的镜像资产 (空间资产-镜像),image_source=custom(仅 image_url + preset 不会关联镜像资产); OBS算法:OBS_ASSETS+algo_path+job_local_path, noalgo_id, Phase 3 skipped. Mandatory; use business names in dialogue. - Input-dataset source (Phase 4) — 按平台数据来源两分(与普通 proc-task 交互
references/interactive-workflow.md§8b/8c 一致):空间资产-数据(BUILD_IN_ASSET, 引用工作空间数据资产 — ① 自动新建数据集资产(create-asset type=dataset+create-version+update-version RELEASE)或 ② 复用已有数据资产,one → default / multiple → multi-select ≤10)与 OBS存储(UDF_OBS_ASSET,③ 直接输入obs://路径,dataset_configsentry 仅obs_path+dataset_type— 不创建、不引用数据资产)。 All ever-asked, never inferred(③ = 平台"OBS存储"选项,与普通 proc-task §8c 行为一致)。 ⚠️ ③ 的自定义obs://路径必须与当前工作空间绑定的 OBS 路径一致:bucket 必须等于cloudrobo workspace current返回的default_obs_path的 bucket(仅支持工作空间绑定的 该 OBS bucket,不支持其它 OBS bucket);交互时先展示default_obs_path作为参考, 校验 bucket 不一致则拒绝并再次确认/重问。 多数据源(多路径/多选数据集,≤10)必须合并为dataset_configs多 entry,只创建 1 条任务, 绝不按数据源拆分任务 —— 仅当用户显式要求批量(每条数据一条任务)且先确认 N 条拆分后才允许 多任务(Mode A/B 一致)。
⚠️ Orchestration flow: the
cloudroboCLI has nooperatorcommand group — the pipeline composes real commands (cloudrobo asset,cloudrobo dataset proc, Docker); CLI/SDK/API equivalence & API paths (audit only) inreferences/dataflow-diagram.md§API Path Summary.
Executable steps live in the operator references (SKILL.md keeps only this summary):
- Custom Operator Workflow — Phase 0-5 steps, phase boundaries, failure-pattern matrix (P0-P3), failure analysis engine, OBS sharded upload, cleanup & report, progress heartbeat
- Custom Operator Reference — operator_config catalog, Mode A/B, dialogue control layer, report template
Key constraints (authoritative detail: workflow §Phase 3/4 + §Failure Analysis Guide):
- Asset names = the dialogue-confirmed operator name (no silent rename / auto suffix; verify via
show-asset). - Image asset FIRST, algorithm asset SECOND; the algorithm MUST link the image asset via
engine.asset_id+engine.asset_version_id+engine.image_source=custom(image_urlalone is NOT linked); never pass--sub-typewith--tags; asseturlis an OBS path. envsMUST exclude platform-injectedSOURCE_DATASET_PATH; Mode Balgo_entranceis absolutepython3 {job_local_path}/entrypoint.py.- New dataset version MUST be
RELEASE; every name /output_nameunique per run (timestamp suffix; duplicates →400 The asset name is existed). - Mode A registers a new image every submission; re-submission =
create-version+update-asset(never re-create-asset); create tasks without--wait, poll. - Mode A re-run after a FAILED verification ⇒ re-submission branch; before
create-taskshow-assetthe algorithm points at the new image version + re-validate dataset sources.create-taskHTTP 500 after a rebuild ⇒ stale linkage or stuckDRAFToutput asset — re-point, freshoutput_name, wait ~30 s, retry once; still 500 →trace_id+ support (details: workflow §Failure Analysis Guide pattern 5d). - Output layout: single-dataset conversion writes flat into the output root (no nested
{output_name}subfolder; multi-dataset{dataset_index}_prefix only); no.cloudrobo-tmpfolder left in the output.
Interaction Guidelines 交互规范
对话逻辑唯一权威定义在 references/custom-operator/custom-operator-reference.md(System
Prompt 交互控制层、S1–S16 应答库、测试用例集),话术冲突以该文件为准。核心规则:一次只问
一个;先展示后询问;模糊即确认(Mode A/B、空间资产-算法/OBS算法、数据集来源①②③、上传方式、镜像 URL 必须
重问——替用户臆断会错配资产注册与任务类型,返工代价高);高危操作二次确认(docker push、OBS 上传、create-asset/version/update、create-task、
删除类);越域拒绝 + 免责;"继续"从上次成功阶段恢复(--reuse-workdir)。
Reference Documents
- ROS Manifest 2.0 — ROS-only field contract, mixed actions, and validation rules
- Bag to Manifest — Local ROS bag profiling, terminal conversation, and Web workbench
- CLI Installation Guide — cloudrobo CLI installation and configuration
- IAM Policies — Least-privilege credential model
- Verification Method — Verification method details
- Dataflow Diagram — Mermaid data flow diagram
- Acceptance Criteria — Acceptance criteria
- Task Config Reference — Algorithm field mapping, required fields template, envs format, eval-tasks field differences
- Interactive Workflow — Detailed step-by-step interactive creation workflows for proc-tasks and eval-tasks
- Edge Cases — full edge-case matrix (task logs, resources, operator rules)
- Custom Operator Workflow — five-phase onboarding + phase boundaries + failure patterns (incl. 5d HTTP 500) + OBS sharded upload + cleanup + progress heartbeat
- Custom Operator Reference — operator_config catalog + Mode A/B + dialogue control layer
Edge Cases
Hard rules that apply to every operation (full edge-case matrix incl. custom-operator items: references/edge-cases.md):
| Scenario | Handling |
|---|---|
Missing workspace_id | Run cloudrobo workspace use or use --workspace-id |
| Task in non-terminal state | wait-task (proc or eval) polls until SUCCEEDED/FAILED/DELETED or timeout |
wait-task timeout | Task may still be running; re-run wait-task and report progress — do not create a duplicate task |
| Log file not found | Falls back to system-std-output.log / job-std-output.log |
| Large log file | Default 64KB tail; use --all or download-log |
| Task creation fails | Check algo_type, JSON validity, dataset_configs, resource quota |
| AK/SK not set | Set HUAWEI_CLOUD_AK/HUAWEI_CLOUD_SK |
| Algorithm not found | Use --name fuzzy search or sub_type filter |
| eval-task delete | Single delete only; no batch |
| proc-task delete | CLI: one task per call; batch via SDK delete_tasks([...]) / API ?ids= |
| Re-run a task | copy-task (neither proc nor eval supports restart) |
| Task events | get-events --task-id <id> --job-name <name> (proc/eval) |
| Pipeline input | For eval, use proc exact asset/version/path in both top-level fields and single-item dataset_configs |
| Evaluation report link | OBS temp URL; save/download promptly |
envs format | JSON array string, not object; see task-config-catalog.md |
catalog_id mismatch | Use workspace's, not algorithm's |
list-assets params | Provide --catalog-id or --repository-id; get from cloudrobo workspace current |
| Algorithm source | Built-in: list-publication-assets; custom: list-assets |
dataset_configs | Use asset's url field; don't manually concatenate |
| Log query | Get file list first for file_path, then fetch content |
| Task states | Terminal: SUCCEEDED, FAILED, DELETED; non-terminal: CREATING, RUNNING, PENDING, FROZEN |
| Task deletion | Irreversible |
| Algorithm fields | Extract from ext_metadata; don't fabricate |
| Object storage | Must use obs://; s3:// prohibited; user-provided OBS paths (input/output) must use the workspace-bound bucket — the bucket of default_obs_path from cloudrobo workspace current; other OBS buckets not supported; mismatch → reject & re-ask |
| API paths | From SDK _url() calls, not inferred |
| Cross-skill | No training/inference; use cloudrobo-train/infer; standalone sim evaluation → huawei-cloud-cloudrobo-eval |
| Cross-skill (pipelines) | Multi-stage raw→conversion→cleaning→evaluation→TrainingReady pipelines (pipeline.yaml/templates) → huawei-cloud-cloudrobo-data-pipeline; this skill covers single proc/eval task operations (CRUD, logs, preview, diagnosis) and a simple proc→eval chain |
| Mutating ops | create/copy/update/delete require confirmation |
| Workflow triggers | No params → interactive; complete params → skip to confirm |
| Missing optional fields | Skip if no environment_variables; use defaults for worker_spec; never send the removed robot_config; omit inputs unless a required input has no built-in url_path or the user asks for a custom mount |
| Empty field validation | Except description, all params must be non-empty; SDK rejects empty strings/None/empty dicts before submission |
head_spec/worker_spec | SDK validation requires cpu and memory; include gpu/npu (value 0 is valid) |
dataset_configs empty | Must contain at least 1 dataset entry; empty array "[]" is invalid |
| eval task-config fields | Use matching top-level dataset fields and a single-item dataset_configs JSON string with exact asset version_id |
| Advanced resources | DEDICATED_POOL: query for IDs; Ray: pair cluster/framework types, non-zero head_spec |
| Custom operator 必选点 | Deploy mode A/B, Mode B task-creation way (空间资产-算法 = WORKSPACE_ASSETS + algo_id — Phase 3 asset link, algo_path = asset code_dir, job_local_path, absolute algo_entrance; OBS算法 = OBS_ASSETS + algo_path + job_local_path, no algo_id), dataset source ①②③ — always ask, never infer (guessing mis-routes Phase 3 asset registration and the task's algo_type) |
| Custom operator 取消/重发 | Phase 4 任务已创建后取消或改参重发:必须先确认删除/停止旧任务 — 展示 {task_id}(运行中)→ 问「①删除(不可恢复)②保留 ③查看日志」→ 确认①后 delete-task + show-task 验证 DELETED;未确认删除前禁止创建新任务(边界 4.11 / S12 / N13) |
Verification Method
Specification Compliance Verification
bash scripts/test-cli-commands.sh -s . -e cli
Functional Testing
# CLI / SDK
bash scripts/test-cli-commands.sh -s . -e {cli|sdk}
API-level verification is manual — replay the API paths in
references/task-config-catalog.md in
API Explorer.
Test Cases
See templates/test-vars.json for the full test case list covering proc-tasks, eval-tasks,
algorithm discovery, pipeline, and custom-operator onboarding scenarios.
Verification Checklist
- After creating a task, poll status (recommended 15s interval), report each state change to user
- On SUCCEEDED: verify output via
get-preview, output system log - On FAILED: ask user whether to view logs, fetch system and job logs per the two-step flow (list files → get content)
- For pipeline: verify processing exact asset/version/path matches eval top-level fields and
dataset_configs[0], then eval SUCCEEDED → get-preview returns OBS link - For eval tasks: verify report preview after SUCCEEDED
- For any task: query
get-events --job-name <name>to surface warnings/errors - For custom-operator onboarding: assets
status=available, Mode Aimage_url/ Mode Bcode_dir,[DONE]log + non-empty frames, user's original code unmodified (details inreferences/verification-method.md)
Best Practices
- Always discover operators first via
cloudrobo asset list-publication-assets --type algorithmbefore creating a task - Use
--dry-runwithcreate-task/copy-taskto validate task config before actual execution - Use
wait-taskwith--timeoutto avoid indefinite polling; report each status transition - On task failure, retrieve both system logs (
--is-system true) and job logs (--is-system false), analyze error patterns (OOM / path not found / image pull failure / OBS access) and suggest fixes before re-running viacopy-task - For pipeline orchestration, wait for processing SUCCEEDED before creating evaluation task
- Before batch delete, confirm task IDs to avoid irreversible deletion
- Clean up completed/failed tasks with
delete-taskto free resources - Set
CLOUDROBO_DEBUG=1for verbose error output during troubleshooting - Evaluation report links are OBS temporary URLs — save or download promptly
- Custom-operator: ask deploy mode A/B, Mode B task-creation way (空间资产-算法 / OBS算法), and dataset source ①②③ FIRST — never infer; multiple data sources (≤10) → ONE task (
dataset_configsN entries, singlecreate-task; only an explicit batch request justifies per-dataset tasks after confirmation); cancel/re-issue after a task is created ⇒ first confirm deletion of the old task (delete-task+ verifyDELETED), never silently create a new one; always--dry-run; cleanup scope = only this run's artifacts (never customer images/containers/data). Full rules:references/custom-operator/custom-operator-workflow.md+references/custom-operator/custom-operator-reference.md
- Downloads
- Security audit
- Review
- Last updated
- 2w ago
- Current version
- v1.0.3
- License
- MIT-0
