huawei-cloud-ecs-shutdown-experiment

Full lifecycle ECS shutdown fault injection experiment for Huawei Cloud chaos engineering: prepare → execute → analyze. Phase 1 discovers target ECS instances, validates compatibility, generates experiment configuration. Phase 2 executes BatchStopServers shutdown, polls status, holds for duration, rolls back via BatchStartServers, verifies recovery, generates execution report. Phase 3 collects CES monitoring metrics and LTS application logs, analyzes error patterns, generates analysis report. Key safety: --dry-run, --auto-rollback, mandatory --yes confirmation, independent emergency rollback script. Triggers: ECS故障演练, ECS关机实验, chaos engineering, 故障注入, ECS shutdown experiment, 演练准备, 执行实验, 运行实验, 启动演练, 执行 ECS 关机故障演练, 分析应用日志, 查看应用表现, 应用日志分析, 故障影响分析, experiment prepare, experiment execute, analyze app logs, log analysis.

huaweicloudskill@huaweicloudskill

Install

openclaw skills install @huaweicloudskill/huawei-cloud-ecs-shutdown-experiment

Huawei Cloud ECS Shutdown Chaos Experiment

⚠️ This skill performs real ECS shutdown operations in Phase 2. Ensure target instances can tolerate downtime. Emergency rollback is always available.

Overview

Full lifecycle ECS shutdown fault injection experiment for Huawei Cloud chaos engineering, covering three phases:

Phase	Name	Action	Key Output
1	Prepare	Discover, validate, generate config	`experiment.json` + `rollback_experiment.sh`
2	Execute	Shutdown → hold → rollback → verify	`execution-log.json` + `execution-report.md`
3	Analyze	Collect CES metrics + LTS logs, analyze patterns	`log-analysis-result.json` + `log-analysis-report.md`

Scope: standalone ECS instances in any availability zone (non-CCE).

Prerequisites

hcloud CLI (KooCLI) — installed and configured with AK/SK. See references/cli-installation-guide.md.
Python 3 — for discovery, validation, execution, and analysis scripts.
Environment variables: HW_ACCESS_KEY, HW_SECRET_KEY, HW_REGION_NAME (optional, default cn-north-4).
IAM permissions: varies by phase — see references/iam-policies.md for the full matrix.
Phase 3 only: LTS log group ID (find via hcloud LTS ListLogGroups or LTS console).

Environment check (run before any phase):

bash

bash scripts/check_env.sh --cli-region <region>

Phase 1: Prepare

Does NOT start the experiment. Only discovers targets, validates compatibility, and generates configuration files.

Step 1.1: Discover Target ECS Instances

bash

python3 scripts/discover_ecs.py --cli-region <region>
python3 scripts/discover_ecs.py --cli-region <region> --az cn-north-4a --status ACTIVE
python3 scripts/discover_ecs.py --cli-region <region> --name-pattern "web-"

Output: JSON with instance ID, name, status, AZ, flavor, charging mode, and tags.

Step 1.2: Validate Compatibility [CRITICAL GATE]

bash

python3 scripts/validate_targets.py --instances-file instances.json --cli-region <region>

Validation rules (see references/ecs-validation-rules.md for details):

Status must be ACTIVE — can only shut down running instances
Spot/bidding instance warning — may be released instead of stopped
AS group association — shutdown may trigger auto-scaling activity
Single-AZ risk — no cross-AZ redundancy if all targets in one AZ

Decision: errors → stop and fix; warnings only → review with user before proceeding.

Step 1.3: Generate Experiment Configuration

bash

python3 scripts/generate_experiment.py \
    --targets "i-xxx,i-yyy" --cli-region <region> --duration 300 --output-dir ./experiments

Output directory: {timestamp}-ecs-shutdown-{slug}/ containing experiment.json + README.md. See references/experiment-template-guide.md for schema.

Step 1.4: Deploy Experiment

bash

bash scripts/deploy_experiment.sh --experiment-dir ./experiments/{dir}/ --cli-region <region>

Generates rollback_experiment.sh (emergency rollback script) in the experiment directory.

Step 1.5: Review

Experiment is prepared but NOT started. Review experiment.json and README.md, then proceed to Phase 2 when ready.

Phase 2: Execute

Real shutdown operation. Target ECS instances are stopped. Ensure experiment.json from Phase 1 is available.

Step 2.1: Execute the Experiment

bash

# Real execution (--yes is mandatory; default refuses without it)
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --yes

# Dry Run (simulated, no API calls)
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --dry-run

# Auto-rollback when shutdown fails
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --auto-rollback --yes

6-phase execution flow (see references/execution-workflow.md for details):

Phase	Action	Description
1. Pre-check	Query status	All instances must be ACTIVE
2. Shutdown	`BatchStopServers`	Inject the fault
3. Monitor	Poll status	Wait until all reach SHUTOFF
4. Wait	`sleep(duration)`	Hold with stop-condition checks
5. Rollback	`BatchStartServers`	Restore instances
6. Verify	Poll status	Confirm all back to ACTIVE

Output: execution-log.json with full execution timeline and state changes.

Step 2.2: Generate Execution Report

bash

python3 scripts/generate_report.py --log-file ./experiments/xxx/execution-log.json

Report contains: experiment overview, target instances, phase timeline, state changes, conclusions.

Emergency Rollback

If the experiment must be terminated early:

bash

python3 scripts/rollback_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region>
# Or pass instance IDs directly:
python3 scripts/rollback_experiment.py --ids "i-xxx,i-yyy" --cli-region <region>

Phase 3: Analyze

Collect CES monitoring metrics and LTS application logs to understand how applications respond to the infrastructure failure. Supports post-hoc mode (after experiment) and real-time mode (during experiment).

Step 3.1: Load Experiment Context

Auto-detect mode: execution-log.json found → post-hoc; falls back to experiment.json → real-time.

Step 3.2: Run Full Analysis Pipeline

bash

python3 scripts/analyze_logs.py \
    --experiment-dir ./experiments/xxx/ \
    --cli-region <region> \
    --log-group-id <lts-group-id>

# Dry run (verify workflow without API calls)
python3 scripts/analyze_logs.py --experiment-dir ./experiments/xxx/ --dry-run

The pipeline executes steps 3.3–3.6 automatically. See references/analysis-workflow.md for details.

Step 3.3: Collect CES Monitoring Metrics

Queries 6 metrics via hcloud CES ShowMetricData (see references/managed-service-logs.md → CES section):

Namespace	Metric	Label	Requires Agent?
`SYS.ECS`	`cpu_util`	CPU Utilization	No
`SYS.ECS`	`network_incoming_bytes_aggregate_rate`	Network Incoming Rate	No
`SYS.ECS`	`network_outgoing_bytes_aggregate_rate`	Network Outgoing Rate	No
`SYS.ECS`	`disk_read_bytes_rate`	Disk Read Rate	No
`SYS.ECS`	`disk_write_bytes_rate`	Disk Write Rate	No
`AGT.ECS`	`mem_usedPercent`	Memory Usage	Yes

Metrics summarized by phase (pre-shutdown / during-shutdown / post-recovery). Query uses period=300 (5-min aggregation for SYS.ECS).

Step 3.4: Collect Logs (LTS)

Queries LTS log group within experiment time window. --log-group-id is required (find via hcloud LTS ListLogGroups). Missing group / empty result = no LTS data, NOT "application unaffected" — check ICAgent config.

Step 3.5: Analyze Error Patterns

Local pattern matching (no network calls):

Pattern	Severity
ERROR/Exception/Traceback	high
Connection refused	high
Timeout	high
HTTP 5xx	high
Reconnect/retry	medium
Degrade/circuit breaker	medium
Recovery/restored	info

Step 3.6: Generate Analysis Report

bash

python3 scripts/generate_analysis_report.py --result-file ./experiments/xxx/log-analysis-result.json

Report contains: experiment overview, affected instances, LTS log sources, instance state timeline, CES monitoring metrics, error pattern statistics, error event timeline, application behavior assessment, improvement suggestions.

Parameter Confirmation

Before each phase's core operation, confirm with the user:

Phase	What to Confirm	Mandatory?
1 (Generate)	Target instance IDs + names, duration, validation warnings	✅ Yes
2 (Execute)	Experiment name, target count, duration, dry-run flag	✅ Yes — `--yes` required for real execution
3 (Analyze)	Experiment directory, detected mode, time window, LTS log group ID	✅ Yes

Wait for explicit confirmation ("确认" / "confirm" / "ok") before proceeding.

Core Commands

<region> = region, defaults to cn-north-4 (HW_REGION_NAME overrides if set). The placeholder may be omitted at runtime — scripts fall back to the default region.

bash

# Environment check (all phases)
bash scripts/check_env.sh --cli-region <region>

# Phase 1: Prepare
python3 scripts/discover_ecs.py --cli-region <region>
python3 scripts/validate_targets.py --instances-file instances.json --cli-region <region>
python3 scripts/generate_experiment.py --targets "i-xxx,i-yyy" --cli-region <region> --duration 300
bash scripts/deploy_experiment.sh --experiment-dir ./experiments/{dir}/ --cli-region <region>

# Phase 2: Execute
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --yes
python3 scripts/generate_report.py --log-file ./experiments/xxx/execution-log.json
python3 scripts/rollback_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region>  # emergency

# Phase 3: Analyze
hcloud LTS ListLogGroups --cli-region=<region> --cli-output=json  # find log group
python3 scripts/analyze_logs.py --experiment-dir ./experiments/xxx/ --cli-region <region> --log-group-id <id>
python3 scripts/generate_analysis_report.py --result-file ./experiments/xxx/log-analysis-result.json

KooCLI Command Format Standard

All hcloud CLI commands follow this format:

text

hcloud <Service> <Action> --cli-region=<region> --cli-output=json [--param=value ...]

--cli-region=<region>: always explicit. KooCLI 7.2.x requires --param=value syntax (equals sign); space-separated --cli-region cn-north-4 is rejected with [USE_ERROR].
--cli-output=json: all commands, for machine-parseable output.
Batch operations use flat parameters (not --body):
- Stop: --os-stop.servers.1.id=<id> --os-stop.type=SOFT
- Start: --os-start.servers.1.id=<id>
Fetch in batch, filter locally: ListServersDetails with --limit=1000 + paging; no N+1 queries.
Credentials via environment variables: AK/SK passed through HW_ACCESS_KEY / HW_SECRET_KEY, never hardcoded.

Script Reference

Script	Phase	Purpose
`scripts/check_env.sh`	All	Environment verification
`scripts/discover_ecs.py`	1	Discover ECS instances
`scripts/validate_targets.py`	1	Validate target compatibility
`scripts/generate_experiment.py`	1	Generate experiment config
`scripts/deploy_experiment.sh`	1	Deploy experiment (local mode)
`scripts/execute_experiment.py`	2	Main execution script (6-phase flow)
`scripts/monitor_instances.py`	2	Instance status monitoring
`scripts/rollback_experiment.py`	2	Emergency rollback
`scripts/generate_report.py`	2	Generate execution report
`scripts/collect_logs.py`	3	LTS log collection + ECS endpoint query
`scripts/collect_ces_metrics.py`	3	CES monitoring metrics collection
`scripts/analyze_logs.py`	3	Main analysis pipeline orchestrator
`scripts/generate_analysis_report.py`	3	Generate Markdown analysis report

Reference Documents

Document	Phase	Description
`references/cli-installation-guide.md`	All	hcloud KooCLI installation and configuration
`references/iam-policies.md`	All	IAM permissions by phase
`references/ecs-validation-rules.md`	1	Validation rules with rationale
`references/experiment-template-guide.md`	1	experiment.json schema
`references/execution-workflow.md`	2	Detailed execution workflow (per-phase I/O, exceptions)
`references/monitoring-guide.md`	2	Monitoring guide (state machine, polling, CES alarms)
`references/analysis-workflow.md`	3	Detailed analysis workflow
`references/managed-service-logs.md`	3	Managed-service log reference (CES, LTS, audit)
`references/verification-method.md`	All	Verification methodology (all phases)
`references/acceptance-criteria.md`	All	Acceptance criteria (all phases)

Key Safety Features

Dry Run mode — --dry-run simulates the full flow without API calls (Phase 2 & 3)
Auto-rollback — --auto-rollback starts instances back up when shutdown fails
Independent rollback script — rollback_experiment.py is standalone, does not depend on main script state
Full logging — execution-log.json records per-phase timestamps and state changes
Pre-check gate — verifies all instances are ACTIVE before execution
Stop-condition early rollback — CES alarm fires → skip remaining duration → rollback immediately
Mandatory confirmation — real execution requires explicit --yes; otherwise refuses
Environment variable whitelist — child processes receive only necessary vars, not full os.environ

ECS Instance State Machine

text

Stop:    ACTIVE → STOPPING → SHUTOFF
Start:   SHUTOFF → STARTING → ACTIVE
Abnormal: any → ERROR

Limitations

Depends on hcloud CLI with sufficient ECS permissions (BatchStopServers, BatchStartServers, ListServersDetails)
AS group check requires AS API access (skipped if unavailable)
CES SYS.ECS metrics are aggregated at 5-minute granularity (period=300); shorter periods return empty
CES API may be unreachable in sandboxed environments — see references/managed-service-logs.md → CES section
AGT.ECS metrics (memory) require telescope/uniagent Agent installed inside the ECS
LTS collection requires ICAgent shipping logs to a LTS log group; missing group = no data, not "unaffected"
Does not support composite scenarios (e.g., multi-action AZ-level interruption) — focused on single ECS shutdown
In real-time mode instance_timeline is unavailable, so recovery-time analysis is skipped
Error-pattern analysis is based on string matching and may miss unusual log formats