Install
openclaw skills install huawei-cloud-cce-pressure-testHuawei Cloud Cce Pressure Test
Huawei Cloud CCE end-to-end workload pressure testing and performance evaluation using Python SDK dispatcher. Trigger: "pressure test", "压测", "load test", "负载测试", "stress test", "压力测试", "performance test", "性能测试", "k6 test", "k6 压测", "ELB traffic test", "ELB 流量测试", "end-to-end pressure test", "全链路压测", "elasticity evaluation", "弹性评估", "traffic generation", "流量生成"
Huawei Cloud CCE Pressure Test
Trigger keywords: pressure test, 压测, load test, 负载测试, stress test, 压力测试, performance test, 性能测试, k6 test, k6 压测, ELB traffic test, ELB 流量测试, end-to-end pressure test, 全链路压测, elasticity evaluation, 弹性评估, traffic generation, 流量生成
⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.
- The dispatcher script is located at
scripts/huawei-cloud.pywithin the skill directory- All scripts and environment check scripts are inside the skill package. You must use
skill action=execto execute them. Do not run them directly in a shell.- Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
- All paths are relative to the skill directory, which is the directory where this SKILL.md is located.
Overview
This skill runs controlled workload pressure tests on Huawei Cloud CCE. It builds a complete traffic path from a k6 client through ELB to nginx-ingress to workload pods, supports short-connection, keepalive, and ramp traffic models, collects ELB metrics and AOM observability data, evaluates elasticity phases, and generates bilingual Markdown and HTML reports with performance curves.
Architecture: Python dispatcher (scripts/huawei-cloud.py) →
Huawei Cloud Python SDK + Kubernetes client → ELB/CCE/AOM/APM APIs →
Route preparation → Client generation → Traffic run →
Observability collection → Report generation
Key Principle: Preview-first. Read-only checks may run immediately,
but Service/Ingress changes, traffic generation, and workload scaling
require explicit user approval with confirm=true before execution.
Related Skills:
| Skill | Purpose |
|---|---|
huawei-cloud-cce-autoscaling-diagnoser | Diagnose autoscaling failures when HPA or CA does not respond during pressure tests |
huawei-cloud-cce-observability-context-builder | Build observability context when AOM/APM evidence is needed |
huawei-cloud-cce-pod-failure-diagnoser | Diagnose Pod runtime failures during traffic runs |
huawei-cloud-cce-node-failure-diagnoser | Diagnose node-level failures under load |
huawei-cloud-cce-workload-failure-diagnoser | Diagnose workload rollout failures after scaling |
huawei-cloud-cce-auto-remediation-runner | Execute remediation actions when issues are found during tests |
huawei-cloud-cce-cluster-management | Cluster lifecycle, nodepool management, kubeconfig retrieval |
Capabilities:
- End-to-end traffic path: k6 client → ELB → nginx-ingress → workload Service → Pod
- Short-connection (
short), keepalive (keepalive), and staged ramp (ramp) traffic models - Isolated Java sample deployment for lab environments
- ELB creation when no reusable ingress-controller ELB exists
- Route preparation: Service and Ingress creation/patching with preview-first
- k6 client generation with ConfigMap and Job
- Traffic run with replica and Pod count monitoring
- APM Java agent injection for distributed tracing during tests
- Observability collection: ELB metrics, Pod CPU/memory, AOM logs
- Bilingual Markdown and HTML report generation with SVG performance curves
- Elasticity evaluation: baseline vs scaled phase comparison
Prerequisites
1. Python Requirements (MANDATORY)
- Python >= 3.6 installed
- Required packages:
huaweicloudsdkcore,huaweicloudsdkcce,huaweicloudsdkvpc,huaweicloudsdkvpcep,huaweicloudsdkaom,huaweicloudsdkelb,kubernetes,matplotlib - Verify:
python3 --version - Install packages:
pip3 install huaweicloudsdkcore huaweicloudsdkcce huaweicloudsdkvpc huaweicloudsdkvpcep huaweicloudsdkaom huaweicloudsdkelb kubernetes matplotlib
2. Credential Configuration
- Valid Huawei Cloud credentials (AK/SK mode)
- Security Rules:
- 🚫 Never expose AK/SK values in code, conversation, or commands
- 🚫 Never use
echo $HUAWEI_AKorecho $HUAWEI_SKto check credentials - 🚫 Never write credentials to files, logs, or responses
- ✅ Use environment variables:
HUAWEI_AK,HUAWEI_SK,HUAWEI_REGION - ✅ Credentials exist only in the current request call stack and are released after each invocation
- ✅ Prefer IAM users over root account for cloud operations
Configuration Method (Environment Variables Only):
export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4
Additional Variables:
| Variable | Required | Description |
|---|---|---|
HUAWEI_AK | Yes | Huawei Cloud Access Key |
HUAWEI_SK | Yes | Huawei Cloud Secret Key |
HUAWEI_REGION | No | Default region (overrides region param if set) |
HUAWEI_PROJECT_ID | No | Project ID (auto-obtained via IAM API when not set) |
3. IAM Permission Requirements
| API Action | Service | Purpose |
|---|---|---|
| CCE cluster read | CCE | Cluster info, workload inspection, Pod metrics |
| CCE workload read/write | CCE (kubeconfig) | Deploy Java sample, create Service/Ingress, scale replicas |
| CCE addon read | CCE | Check ingress controller and metrics addon |
| ELB create/list/read | ELB | Create load balancer for test traffic path |
| VPC subnet list | VPC | Find candidate subnets for ELB VIP |
| VPCEP endpoint list | VPCEP | Check existing VPCEP endpoints |
| AOM metrics read | AOM | Collect Pod CPU/memory and custom metrics |
| AOM instances list/resolve | AOM | Find AOM instance for the cluster |
| APM master address read | APM | Get APM master address for Java agent injection |
| LTS log query | LTS | Correlate application errors with traffic curves |
Permission Failure Handling:
- When any action fails due to IAM permission errors, display the required permission list
- Guide the user to create custom policies in the IAM console for Huawei Cloud permissions
- Pause execution and wait for user confirmation that permissions have been granted
- Retry the failed action
Core Commands
All actions are invoked via the dispatcher script:
python3 scripts/huawei-cloud.py <action> region=<region> cluster_id=<cluster_id> [key=value ...]
1. Deploy Java Sample (Optional)
# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_deploy_cce_pressure_test_java_sample \
region=cn-north-4 cluster_id=<cluster_id>
# Apply after user approval
python3 scripts/huawei-cloud.py huawei_deploy_cce_pressure_test_java_sample \
region=cn-north-4 cluster_id=<cluster_id> confirm=true
Creates an isolated Namespace, ConfigMap (small Java HTTP server), and Deployment with readiness/liveness probes. Exposes /healthz, /api/hello, and /api/work on port 8080.
2. Create ELB (When No Reusable ELB Exists)
# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_create_elb \
region=cn-north-4 name=<elb-name> vip_subnet_cidr_id=<subnet-network-id>
# Apply after user approval
python3 scripts/huawei-cloud.py huawei_create_elb \
region=cn-north-4 name=<elb-name> vip_subnet_cidr_id=<subnet-network-id> confirm=true
Creates a chargeable ELB. Review billable fields before applying. Does not silently create an EIP.
3. Prepare Route
# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_prepare_cce_pressure_test_route \
region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> workload_name=<deployment-name> target_port=8080
# Apply after user approval
python3 scripts/huawei-cloud.py huawei_prepare_cce_pressure_test_route \
region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> workload_name=<deployment-name> target_port=8080 confirm=true
Creates or patches a ClusterIP Service and Ingress pointing to the workload. Discover an existing nginx ingress controller LoadBalancer address. Does not silently create a chargeable ELB.
4. Inject APM Java Agent (Optional)
python3 scripts/huawei-cloud.py huawei_inject_cce_apm_javaagent \
region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> \
workload_name=<deployment-name> app_name=<apm-app-name> \
business=<apm-business> env_name=<apm-env>
Injects the Huawei Cloud APM Java agent into the workload Deployment for distributed tracing during the pressure test.
5. Generate k6 Client
python3 scripts/huawei-cloud.py huawei_generate_cce_pressure_test_client \
target_url=http://<elb-address>/<path> namespace=pressure-test \
model=keepalive vus=10 duration_seconds=60
Generates the k6 ConfigMap and Job manifest. Use a regional SWR mirror for the k6 image when public image pulls are unavailable.
Supported Traffic Models:
| Model | Purpose |
|---|---|
short | Disable connection reuse; observe new-connection pressure |
keepalive | Reuse connections for a steady HTTP baseline |
ramp | Increase traffic in stages then scale down |
6. Run Pressure Test
# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_run_cce_pressure_test \
region=cn-north-4 cluster_id=<cluster_id> target_url=http://<elb-address>/<path>
# Apply after user approval
python3 scripts/huawei-cloud.py huawei_run_cce_pressure_test \
region=cn-north-4 cluster_id=<cluster_id> target_url=http://<elb-address>/<path> confirm=true
Ensures the client namespace, creates the ConfigMap and Job, waits for completion by default, records replica/Pod count samples, and extracts the k6 summary from logs.
7. Collect Observability
python3 scripts/huawei-cloud.py huawei_collect_cce_pressure_test_observability \
region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> \
workload_name=<deployment-name> elb_id=<elb-id>
Collects ELB QPS, active connections, layer-7 response time, 2xx ratio,
Pod CPU/memory metrics, and AOM evidence. Pass label_selector to scope
Pod metrics to the workload. Pod CPU and memory are collected from node
metrics when aom_instance_id is not provided.
8. Generate Report
python3 scripts/huawei-cloud.py huawei_generate_cce_pressure_test_report \
result_path=<path-to-run-json>
Generates Markdown, HTML, and SVG curve artifacts from the run JSON and optional observability JSON.
9. Scale Workload (Elasticity Evaluation)
# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_scale_cce_workload \
region=cn-north-4 cluster_id=<cluster_id> workload_type=Deployment \
name=<deployment-name> namespace=<namespace> replicas=5
# Apply after user approval
python3 scripts/huawei-cloud.py huawei_scale_cce_workload \
region=cn-north-4 cluster_id=<cluster_id> workload_type=Deployment \
name=<deployment-name> namespace=<namespace> replicas=5 confirm=true
For elasticity evaluation: run a baseline phase, preview scaling, apply only after approval with confirm=true, then run a second phase and compare reports.
10. Supporting Actions (Read-Only)
| Action | Required Params | Description |
|---|---|---|
huawei_list_cce_hpas | region, cluster_id | List HPA specs and conditions |
huawei_get_cce_pods | region, cluster_id | List Pod phase, resources, node assignment |
huawei_get_cce_services | region, cluster_id | List Service types and cluster IPs |
huawei_get_cce_ingresses | region, cluster_id | List Ingress rules and LoadBalancer IPs |
huawei_get_cce_pod_metrics_topN | region, cluster_id | Pod resource metric ranking |
huawei_get_elb_metrics | region, elb_id | ELB QPS, connections, latency, success rate |
huawei_get_elb_backend_status | region, elb_id | Backend member health status |
huawei_list_vpc_subnets | region | List VPC subnets for ELB VIP placement |
huawei_list_aom_instances | region | List AOM instances |
huawei_resolve_cce_aom_instance | region, cluster_id | Resolve AOM instance for a CCE cluster |
huawei_get_apm_master_address | region | Get APM master address for agent injection |
huawei_get_aom_metrics | region, aom_instance_id, query | AOM/Prometheus metric queries |
huawei_query_aom_logs | region, cluster_id | Query AOM LTS logs |
huawei_get_cce_logconfigs | region, cluster_id | List CCE log collection configs |
huawei_create_cce_logconfig | region, cluster_id, logconfig_name, source_type, log_group_id, log_stream_id | Create CCE log collection config |
huawei_get_application_logconfigs | region, cluster_id, app_name | Get application log configs |
huawei_query_application_logs | region, cluster_id, app_name | Query application logs |
huawei_analyze_application_logs | region, cluster_id, app_name | Analyze application logs for errors |
huawei_generate_monitor_dashboard | region, cluster_id | Generate AOM monitor dashboard URL |
Parameter Reference
Common Parameters
| Parameter | Required | Description | Default |
|---|---|---|---|
region | Yes | Huawei Cloud region | - |
cluster_id | Yes (most actions) | CCE cluster ID | - |
namespace | Action-dependent | Kubernetes namespace | - |
workload_name | Action-dependent | Deployment name | - |
ak | No | Access Key (overrides env var) | HUAWEI_AK env |
sk | No | Secret Key (overrides env var) | HUAWEI_SK env |
project_id | No | Project ID (overrides env var) | Auto-resolved |
Deploy Java Sample Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
confirm | Required for mutation | Approve deployment | true to apply, omit to preview |
ELB Creation Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
name | Yes | ELB name | - |
vip_subnet_cidr_id | Yes | VPC subnet network ID for ELB VIP | Obtain from huawei_list_vpc_subnets |
availability_zone_list | No | AZ list | Comma-separated |
l4_flavor_id | No | Layer-4 flavor ID | Optional performance spec |
l7_flavor_id | No | Layer-7 flavor ID | Optional performance spec |
confirm | Required for mutation | Approve ELB creation | true to apply, omit to preview |
Route Preparation Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
target_port | No | Container target port | Defaults to workload's first container port |
host | No | Ingress host rule | Pass same value as host_header to client |
selector_json | No | Pod selector for non-Deployment workloads | JSON string |
confirm | Required for mutation | Approve route creation | true to apply, omit to preview |
Client Generation Parameters
| Parameter | Required | Description | Default |
|---|---|---|---|
target_url | Yes | Full URL to test | - |
namespace | No | k6 client namespace | pressure-test |
model | No | Traffic model (short, keepalive, ramp) | keepalive |
vus | No | Number of virtual users | 10 |
duration_seconds | No | Test duration | 60 |
host_header | No | Host header for Ingress host rules | Matches host from route preparation |
Traffic Run Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
target_url | Yes | Full URL to test | Must match client target |
confirm | Required for mutation | Approve traffic run | true to apply, omit to preview |
Observability Collection Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
elb_id | No | ELB ID for traffic/connection/latency metrics | Critical for full observability |
label_selector | No | Pod label selector | Scope Pod metrics to workload |
aom_instance_id | No | AOM instance ID | Auto-resolved if omitted |
APM Injection Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
app_name | Yes | APM application name | - |
business | Yes | APM business name | - |
env_name | Yes | APM environment name | - |
Report Generation Parameters
| Parameter | Required | Description | Notes |
|---|---|---|---|
result_path | Yes | Path to run JSON file | Combine with observability JSON if available |
Common Region IDs
| Region Name | Region ID |
|---|---|
| North China - Beijing 4 | cn-north-4 |
| North China - Beijing 1 | cn-north-1 |
| East China - Shanghai 1 | cn-east-3 |
| East China - Shanghai 2 | cn-east-2 |
| South China - Guangzhou | cn-south-1 |
| South China - Shenzhen | cn-south-4 |
| Southwest China - Guiyang 1 | cn-southwest-2 |
| Asia Pacific - Bangkok | ap-southeast-2 |
| Asia Pacific - Singapore | ap-southeast-1 |
| Asia Pacific - Hong Kong | ap-southeast-3 |
| Europe - Paris | eu-west-0 |
Output Format
All actions return JSON with the following common structure:
| Field | Description |
|---|---|
success | Boolean: true if operation succeeded, false otherwise |
action | Action name that was executed |
region | Huawei Cloud region |
Run JSON (huawei_run_cce_pressure_test when output_dir is provided):
| Field | Description |
|---|---|
job | Job creation, completion, failure, and timeout state |
samples | Workload replica and Pod count samples collected during the run |
metric_series | Standardized replica curves from the run |
k6_summary | Requests, RPS, success rate, latency percentiles, bytes, and max VUs parsed from k6 logs |
data_gaps | Missing evidence that should be called out in the report |
Observability JSON (huawei_collect_cce_pressure_test_observability):
| Field | Description |
|---|---|
inventory | Pods, Services, and Ingresses in the workload namespace |
pod_metrics | Node-backed Pod CPU and memory metrics |
elb_metrics | ELB QPS, connections, latency, and response metrics when elb_id is provided |
elb_backend_status | Backend member and health-monitor evidence |
metric_series | Standardized curves consumed by the report action |
Report Artifacts (huawei_generate_cce_pressure_test_report):
| Artifact | Purpose |
|---|---|
*-report.md | Portable Markdown report |
*-report.html | Styled HTML report with risk-colored recommendations and data-gap tables |
*-curves.svg | Embedded curve chart for traffic, latency, resources, and replicas |
See Output Schema for the full JSON response schema.
Verification
Step-by-step Verification Checklist
- Verify AK/SK credentials are configured via environment variables
- Inspect existing Services, Ingresses, HPA, ingress controller, and ELB context
- Confirm cluster, workload, namespace, target port, test window, traffic model, VUs, and output directory
- Deploy Java sample (preview → confirm) if needed for an isolated lab
- Create ELB (preview → confirm) if no reusable ingress-controller ELB exists
- Prepare route (preview → confirm) — Service and Ingress
- Inject APM Java agent (optional) for distributed tracing
- Generate k6 client manifest — review ConfigMap and Job
- Run pressure test (preview → confirm)
- Collect observability with
elb_idfor full evidence - Generate report and review Markdown/HTML/SVG artifacts
- For elasticity evaluation: run baseline → scale workload → run second phase → compare reports
Best Practices
- Start with low traffic: Always begin with low VUs and short duration to validate the traffic path before ramping up
- Preview before apply: Always run mutation actions without
confirm=truefirst, review the plan, then re-run withconfirm=trueafter explicit user approval - Use regional SWR images for k6: Docker Hub images may timeout; mirror k6 to a regional SWR namespace
- Pass
elb_idfor observability: ELB metrics (QPS, connections, latency, success rate) are critical evidence; always passelb_idwhen collecting observability - Run at least two phases for elasticity: Baseline + scaled phase comparison is the standard methodology for evaluating autoscaling behavior
- Confirm test target before traffic: Never send traffic to production paths without an approved test window
- Stop raising traffic when indicators degrade: Halt when success rate drops, latency rises sharply, or resource waterlines exceed agreed limits
- Do not delete resources automatically: Never auto-delete ELBs, namespaces, workloads, Jobs, or ConfigMaps
Reference Documents
| Document | Description |
|---|---|
| Workflow | Staged execution, action parameters, and command examples |
| Risk Rules | Preview-first constraints, mutation boundaries, and operational limits |
| Output Schema | JSON response schema and report artifact structure |
Notes
- Preview-first by design — Service/Ingress changes, traffic generation, and workload scaling return a preview without
confirm=true; apply only after explicit user approval - Standard traffic path —
Pod → Service → nginx-ingress → ELB; the route preparation action discovers existing ingress controllers and does not silently create chargeable resources - ELB is chargeable — creating a new ELB incurs hourly billing; always preview and confirm before creation
- No credential persistence — AK/SK exists only during API calls; never written to disk, logs, or reports
- Cross-skill escalation — If autoscaling does not respond during tests, hand off to
huawei-cloud-cce-autoscaling-diagnoser; if Pods fail, hand off tohuawei-cloud-cce-pod-failure-diagnoser
Common Pitfalls
| Pitfall | Symptom | Quick Fix |
|---|---|---|
| Public Docker Hub k6 image | Image pull timeout in CCE | Mirror k6 image to regional SWR namespace |
| Missing ELB in traffic path | k6 client cannot reach workload | Create ELB or verify ingress-controller LoadBalancer IP |
| Ingress host mismatch | k6 requests rejected with 404 | Pass host during route prep and host_header to client |
No observability without elb_id | Missing traffic/latency curves in report | Always pass elb_id to observability collection |
| Skipping baseline phase | Cannot evaluate elasticity improvement | Run baseline phase before scaling |
Wrong vip_subnet_cidr_id | ELB creation fails | Use network ID from huawei_list_vpc_subnets, not subnet UUID |
| Production traffic without approval | Unintended load on production | Confirm test target and namespace before traffic generation |