huawei-cloud-cloudrobo-dataset

Manage CloudRobo data processing (proc-tasks) and data evaluation (eval-tasks) — query algorithms, create/list/show/update/delete/restart tasks; poll task status until terminal; retrieve system and job logs; download logs; preview output data and frames; discover available algorithms from the asset marketplace; orchestrate processing→evaluation pipelines; batch-manage tasks across workspaces; diagnose failures via log analysis. Triggers include: data processing task management, data evaluation task management, task status polling, log retrieval for troubleshooting, output data preview, algorithm discovery, task pipeline orchestration, batch task management, failure diagnosis, dataset, proc-tasks, eval-tasks.

huaweicloud-skills-team@huaweiclouddev

Install

openclaw skills install @huaweiclouddev/huawei-cloud-cloudrobo-dataset

Windows / PowerShell: Examples use bash syntax. To run on Windows PowerShell:

Flatten \ line continuations to a single line, or end lines with a backtick.

Set env vars with $env:NAME="value" instead of export NAME="value".

Single-quoted JSON '{"a":"b"}' works as-is.

Overview

The cloudrobo-dataset skill manages the full lifecycle of CloudRobo data processing (proc-tasks) and data evaluation (eval-tasks). It covers algorithm query, task CRUD operations, status polling, log retrieval (system + job) and download, output data preview, frame extraction, algorithm discovery from the asset marketplace, processing→evaluation pipeline orchestration, batch task management, and failure diagnosis.

Applicable scenarios: Task management (CRUD + restart), troubleshooting (log retrieval + error analysis), algorithm discovery, result verification (preview + frames), pipeline orchestration, batch management, analytics.

Architecture:

text

Agent / LLM
    │
    ├── CLI  →  cloudrobo dataset proc <command>     (proc-tasks)
    │           cloudrobo dataset eval <command>     (eval-tasks)
    ├── SDK  →  DatasetClient (Python)
                    │
                    ▼
              cloudrobo-service (REST API)
              /v1/data-eng/proc-tasks/*
              /v1/data-eng/eval-tasks/*

All operations target the cloudrobo-service backend and require a workspace_id (default workspace is used unless --workspace-id overrides it). Algorithm and dataset discovery is a cross-package operation that calls the cloudrobo-asset-manager service via cloudrobo asset.

Prerequisites

See references/cli-installation-guide.md for CLI installation, AK/SK authentication, and workspace configuration. All dataset operations require a valid workspace_id (default or explicit --workspace-id).

Workflow

Interactive Task Creation

All interactive creation workflows follow the Interaction Rules: at most one tool call per turn, display query results before asking the next question.

Proc-Task Creation (10 steps): Task name → Description → Algorithm type (PRESET_ASSETS/WORKSPACE_ASSETS/OBS_ASSETS) → Algorithm configuration with field mapping → Environment variables → Job type and resources (CCE/K8S or CCE_RAY/RAY, resource pool, instance spec) → Dynamic storage (public pool only) → Input datasets (multi-dataset loop) → Output configuration → Confirm and submit. See references/interactive-workflow.md for detailed steps.

Eval-Task Creation (7 steps): Task name → Description → Resource pool and instance spec → Evaluation algorithm → Dataset (single) → Robot config → Confirm and submit. See references/interactive-workflow.md for detailed steps.

Task Polling and Result Handling

Poll status — dataset proc wait-task (recommended 15s interval), report each state change to user, until terminal state (SUCCEEDED/FAILED/DELETED)
On completion:
- SUCCEEDED: view processing result via get-preview, output system log
- FAILED: ask user whether to view logs, output system and job logs

Eval tasks: After SUCCEEDED, extract report path from target_report_path in task detail, use dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> to get OBS temporary link.

Pipeline Workflow (Task Orchestration)

Scenario: data conversion → data evaluation → download artifact

Run processing task, wait for SUCCEEDED
Extract target_path and target_asset_id from task detail
Construct dataset_configs entry with obs_path = target_path, asset_id = target_asset_id
Create evaluation task, wait for SUCCEEDED
Get report: dataset eval get-preview --file-name <path-from-target_report_path> (remove obs://<bucket>/ prefix)
Download artifacts via download-asset

Batch Workflow (Batch Tasks)

Scenario: run same algorithm across multiple datasets

Get dataset list (asset list-assets --type dataset)
Create processing task per dataset (reuse algorithm/resource config from first; only input dataset differs)
Collect task_ids, poll status
Summarize success/failure/in-progress counts
For failed tasks: view logs or restart

Diagnosis Workflow (Failure Diagnosis)

Scenario: task FAILED → analyze logs, suggest fixes

Get system and user logs
Analyze error patterns: OOM → increase worker_spec.memory; path not found → check dataset_configs; image pull failure → check image; parameter error → check algo_entrance/envs; access denied → check catalog_id/permissions
Output diagnosis and fix suggestions
After confirmation, modify via update-task or restart via restart-task

Analytics Workflow (Data Analysis)

Aggregate task stats by status, algorithm ranking, time trend, failure rate. Output analysis report with root-cause suggestions.

Long-Running Task Workflow

Start wait-task with timeout. On timeout, report status and elapsed time; user decides: continue / view logs / terminate.

CLI Command Format Standard

bash

cloudrobo dataset proc <command> [OPTIONS]
cloudrobo dataset eval <command> [OPTIONS]

Feature	Description	Example
Command group	`dataset proc` (proc-tasks), `dataset eval` (eval-tasks)	`cloudrobo dataset proc`
Subcommand	kebab-case	`create-task`, `list-tasks`, `show-task`
Workspace override	`--workspace-id <id>`	`--workspace-id abc-123`
Output format	JSON to stdout	`out(result)`
Dry-run	`--dry-run` (where supported)	Preview without executing
Boolean flag	`--is-system true`	`--is-system true`
Comma list	`--task-ids id1,id2,id3`	`--task-ids aaa,bbb`

Core Commands

Task Management (proc-tasks)

Create a data processing task

bash

cloudrobo dataset proc create-task --name <task-name> --algo-type PRESET_ASSETS --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800] [--dry-run]

SDK: client.create_task(task_config: dict)
API: POST /v1/data-eng/proc-tasks

Required task_config fields: name, algo_type (PRESET_ASSETS / WORKSPACE_ASSETS / OBS_ASSETS, based on Step 3 selection), algo_name, algo_entrance, image, algo_id, catalog_id (workspace's), resource_pool_type, cluster_type, task_framework_type, dataset_configs (JSON string array, multi-dataset input), output_type, output_path, output_name, head_spec (must include cpu/memory/gpu/npu), worker_spec (must include cpu/memory/gpu/npu), worker_num, evs_spec. See references/task-config-catalog.md for full field mapping.

⚠️ Non-Empty Validation: Except description, all parameters cannot be empty values (empty string "", None, empty dict {}). The SDK validates this before API submission. head_spec/worker_spec must include all four keys: cpu, memory, gpu, npu (value 0 is valid). dataset_configs must contain at least 1 dataset entry; empty array "[]" is invalid.

List tasks

bash

cloudrobo dataset proc list-tasks [--status RUNNING|SUCCEEDED|FAILED] [--algo-type <type>] [--name <name>] [--order-by start_at|update_at|finish_at] [--order DESC|ASC] [--offset <n>] [--limit <n>] [--user-id <id>] [--algo-name <name>] [--output-name <name>] [--workspace-id <id>]

SDK: client.list_tasks(workspace_id=None, statuses=..., algo_type=..., name=..., order_by=..., ...)
API: GET /v1/data-eng/proc-tasks?workspace_id=<id>&statuses=<status>

Show task detail

bash

cloudrobo dataset proc show-task --task-id <task-id>

SDK: client.get_task_detail(task_id)
API: GET /v1/data-eng/proc-tasks/{task_id}

Update a task (SDK only)

SDK: client.update_task(task_id, task_config: dict)
API: PATCH /v1/data-eng/proc-tasks/{task_id}

Delete tasks (SDK only)

SDK: client.delete_tasks([task_id_1, task_id_2])
API: DELETE /v1/data-eng/proc-tasks?ids=id1,id2

Restart a task

bash

cloudrobo dataset proc restart-task --task-id <task-id>

SDK: client.restart_task(task_id)
API: POST /v1/data-eng/proc-tasks/{task_id}/restart

Task Monitoring

Wait for task to reach terminal state

bash

cloudrobo dataset proc wait-task --task-id <task-id> [--timeout 1800] [--interval 10]

Terminal states: SUCCEEDED, FAILED, DELETED

SDK: client.wait_task(task_id, timeout=1800, interval=10, on_status=callback)
API: GET /v1/data-eng/proc-tasks/{task_id} (polled)

Retrieve task logs

bash

# Step 1: List log files to get file_path
cloudrobo dataset proc get-log --task-id <id> --is-system true
cloudrobo dataset proc get-log --task-id <id> --is-system false

# Step 2: Get log content (default: latest 64KB tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path>

# Get full log (not just tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path> --all

SDK: client.list_log_files(task_id, is_system=True) → client.get_task_log(task_id, file_name, file_path, start_byte, end_byte) or client.get_task_log_tail(task_id, file_name)
API:
- GET /v1/data-eng/proc-tasks/{task_id}/logs?is_system=true|false
- GET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}?start_byte=&end_byte=&file_path=&job_id=

Download a log file

bash

cloudrobo dataset proc download-log --task-id <id> --file-name <name> --file-path <path>

SDK: client.download_task_log(task_id, file_name, file_path)
API: GET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}/download

Get task resource usage

bash

cloudrobo dataset proc get-resource-usage --task-id <task-id> --metric CPU_UTIL|CPU_USED_CORE|MEM_UTIL|MEM_USED_MB|NETWORK_TX_RATE|NETWORK_RX_RATE|DISK_READ_KB|DISK_WRITE_KB --start <unix-ts-sec> --end <unix-ts-sec> --step <10-3600>

SDK: client.get_task_resource_usage(task_id, metric, start, end, step)
API: GET /v1/data-eng/proc-tasks/{task_id}/resource-usage?metric=&start=&end=&step=

Data Preview

Preview task output data

Get OBS temporary download link for a dataset file. file_name = OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../data/chunk-000/file-000.parquet).

bash

cloudrobo dataset proc get-preview --task-id <task-id> --file-name <obs-path-without-bucket>

SDK: client.get_task_preview(task_id, file_name)
API: GET /v1/data-eng/proc-tasks/{task_id}/preview?file_name=<file_name>

Get task frames

Query directory file list of task input/output datasets. prefix = dataset OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../).

bash

cloudrobo dataset proc get-frames --task-id <task-id> --prefix <obs-path-without-bucket>

SDK: client.get_task_frames(task_id, prefix)
API: GET /v1/data-eng/proc-tasks/{task_id}/frames?prefix=<prefix>

Evaluation Tasks (eval-tasks)

Evaluation tasks differ from processing tasks in field names, deletion granularity, and supported operations. See the full comparison table in references/task-config-catalog.md.

Create an evaluation task

bash

cloudrobo dataset eval create-task --name <task-name> --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800]

SDK: client.create_eval_task(task_config: dict)
API: POST /v1/data-eng/eval-tasks

task-config field description:

The eval-task task-config requires two sets of dataset fields: top-level single-value fields (dataset_type, dataset_id, dataset_name, dataset_path) + dataset_configs array (single dataset, format same as proc-task).

Required fields: name, algo_type, algo_id, algo_name, algo_entrance, image, catalog_id (workspace's), cluster_type, task_framework_type, dataset_type/dataset_id/dataset_name/dataset_path (top-level), dataset_configs, robot_config, resource_pool_type, head_spec, worker_spec, worker_num, evs_spec, output_type, output_path, output_name.

Optional: description, resource_id, dedicated_pool_name.

⚠️ Key reminders:

Both top-level dataset fields and dataset_configs array required
dataset_name cannot be empty (extract from OBS path last directory for UDF_OBS_ASSET)
dataset_type values: BUILD_IN_ASSET / UDF_OBS_ASSET

See references/task-config-catalog.md for full field mapping and eval-task field differences.

List evaluation tasks

bash

cloudrobo dataset eval list-tasks [--status <status>] [--name <name>] [--workspace-id <id>]

SDK: client.list_eval_tasks(workspace_id=None, **params)
API: GET /v1/data-eng/eval-tasks?workspace_id=<id>

Show evaluation task detail

bash

cloudrobo dataset eval show-task --task-id <task-id>

SDK: client.get_eval_task_detail(task_id)
API: GET /v1/data-eng/eval-tasks/{task_id}

Update an evaluation task (SDK only)

SDK: client.update_eval_task(task_id, task_config: dict)
API: PATCH /v1/data-eng/eval-tasks/{task_id}

Delete an evaluation task (single granularity, SDK only)

SDK: client.delete_eval_task(task_id)
API: DELETE /v1/data-eng/eval-tasks/{task_id}

Note: eval-tasks deletion is single-task granularity (delete-task), same as proc-tasks (delete-task). eval-tasks do not support restart.

Retrieve evaluation task logs

bash

# Step 1: List log files
cloudrobo dataset eval get-log --task-id <id> --is-system true
cloudrobo dataset eval get-log --task-id <id> --is-system false

# Step 2: Get log content
cloudrobo dataset eval get-log --task-id <id> --file-name <name> --file-path <path>

SDK: client.list_eval_log_files(task_id, is_system=True) + client.get_eval_task_log(task_id, file_name, ...)
API:
- GET /v1/data-eng/eval-tasks/{task_id}/logs?is_system=true|false
- GET /v1/data-eng/eval-tasks/{task_id}/logs/{file_name}?file_path=&job_id=

Get evaluation task preview

file_name is the OBS path of the report file (excluding bucket name). Extract from task detail's target_report_path by removing obs://<bucket-name>/ prefix.

For example: if target_report_path is obs://cloudrobo-test-203/eval_task/report/abc-123/diversity_evaluation_report.pdf, then file_name = eval_task/report/abc-123/diversity_evaluation_report.pdf.

bash

cloudrobo dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> [--is-download]

SDK: client.get_eval_task_preview(task_id, file_name, is_download=False)
API: GET /v1/data-eng/eval-tasks/{task_id}/preview?file_name=<obs-path-without-bucket>&isDownload=

Returns OBS temporary URL. Use --is-download for download link, omit for preview link. Save or download promptly.

Algorithm Discovery

List available algorithms

bash

cloudrobo asset list-publication-assets --type algorithm --tags "Data Processing" [--name <fuzzy-name>] [--limit 20]

SDK (cross-package): asset_client.list_publication_assets(type="algorithm", sub_type="data_processing", limit=20)
API: Cross-package — calls the asset service, not the dataset service directly.

Each algorithm includes ext_metadata with engine.image_url, command, and environment_variables needed for task creation. Use --tags "Data Evaluation" to discover evaluation algorithms.

Reference Documents

CLI Installation Guide — cloudrobo CLI installation and configuration
IAM Policies — Least-privilege credential model
Verification Method — Verification method details
Dataflow Diagram — Mermaid data flow diagram
Acceptance Criteria — Acceptance criteria
Task Config Reference — Algorithm field mapping, required fields template, envs format, eval-tasks field differences
Interactive Workflow — Detailed step-by-step interactive creation workflows for proc-tasks and eval-tasks

Edge Cases

Scenario	Handling
Missing `workspace_id`	Run `cloudrobo workspace use` or use `--workspace-id`
Task in non-terminal state	`wait-task` polls until SUCCEEDED/FAILED/DELETED or timeout
Log file not found	Falls back to `system-std-output.log` / `job-std-output.log`
Large log file	Default 64KB tail; use `--all` or `download-log`
Task creation fails	Check algo_type, JSON validity, dataset_configs, resource quota
AK/SK not set	Set `HUAWEI_CLOUD_AK`/`HUAWEI_CLOUD_SK`
Algorithm not found	Use `--name` fuzzy search or `sub_type` filter
eval-task delete	Single delete only; no batch, no restart
Pipeline input	Wrap `target_path` into `dataset_configs` entry
Evaluation report link	OBS temp URL; save/download promptly
`envs` format	JSON array string, not object; see task-config-catalog.md
`catalog_id` mismatch	Use workspace's, not algorithm's
`list-assets` params	Provide `--catalog-id` or `--repository-id`; get from `cloudrobo workspace current`
Algorithm source	Built-in: `list-publication-assets`; custom: `list-assets`
`dataset_configs`	Use asset's `url` field; don't manually concatenate
Log query	Get file list first for `file_path`, then fetch content
Task states	Terminal: SUCCEEDED, FAILED, DELETED; non-terminal: CREATING, RUNNING, PENDING, FROZEN
Task deletion	Irreversible
Algorithm fields	Extract from `ext_metadata`; don't fabricate
Object storage	Must use `obs://`; `s3://` prohibited
API paths	From SDK `_url()` calls, not inferred
Cross-skill	No training/inference; use cloudrobo-train/infer
Mutating ops	create/update/delete/restart require confirmation
Workflow triggers	No params → interactive; complete params → skip to confirm
Missing optional fields	Skip if no `environment_variables`; use defaults for `worker_spec`; prompt for `robot_config` in eval
Empty field validation	Except `description`, all params must be non-empty; SDK rejects empty strings/None/empty dicts before submission
`head_spec`/`worker_spec`	Must include `cpu`, `memory`, `gpu`, `npu` keys (value `0` is valid)
`dataset_configs` empty	Must contain at least 1 dataset entry; empty array `"[]"` is invalid
Advanced resources	DEDICATED_POOL: query for IDs; Ray: pair cluster/framework types, non-zero head_spec

Verification Method

Specification Compliance Verification

bash

bash scripts/test-cli-commands.sh -s . -e cli

Functional Testing

bash

# CLI / SDK
bash scripts/test-cli-commands.sh -s . -e {cli|sdk}

API-level verification is manual — replay the API paths in references/task-config-catalog.md in API Explorer.

Test Cases

See templates/test-vars.json for the full test case list covering proc-tasks, eval-tasks, algorithm discovery, and pipeline scenarios.

Verification Checklist

After creating a task, poll status (recommended 15s interval), report each state change to user
On SUCCEEDED: verify output via get-preview, output system log
On FAILED: ask user whether to view logs, fetch system and job logs per the two-step flow (list files → get content)
For pipeline: verify processing target_path is wrapped into dataset_configs for the eval task, then eval SUCCEEDED → get-preview returns OBS link
For eval tasks: verify report preview after SUCCEEDED

Best Practices

Always run list-algorithms first to discover available operators before creating a task
Use --dry-run with create-task to validate task config before actual execution
Use wait-task with --timeout to avoid indefinite polling; report each status transition
On task failure, retrieve both system logs (--is-system true) and job logs (--is-system false), analyze error patterns (OOM / path not found / image pull failure / OBS access) and suggest fixes before restarting
For pipeline orchestration, wait for processing SUCCEEDED before creating evaluation task
Before batch delete, confirm task IDs to avoid irreversible deletion
Clean up completed/failed tasks with delete-task to free resources
Set CLOUDROBO_DEBUG=1 for verbose error output during troubleshooting
Evaluation report links are OBS temporary URLs — save or download promptly