huawei-cloud-dcs-failover

Huawei Cloud DCS master-slave failover (AZ power-off) drill tool. One-click pipeline: prepare (checks + risk assessment) -> execute (failover) -> report. Queries DCS master-slave instance, checks IAM permissions, architecture hard-gate (ha + cross-AZ + RUNNING), safely executes master-slave failover (ChangeMasterStandby), polls for completion, data consistency check, three-dimensional scoring (failover/data/business, written to execution_result.json), generates HTML report. prepare phase includes built-in risk assessment (instance health, master-slave nodes, memory/CPU/connections/latency load, backup safety net, etc.). Critical risks block failover by default (user must confirm to proceed). Supports Redis master-slave instances. Triggers: DCS主备倒换, 主备倒换演练, DCS故障演练, 主备切换, 倒换实验, dcs failover, DCS failover, 高可用演练, 高可用验证, DCS HA切换, AZ断电演练, 主备互换, DCS switchover, Redis故障演练, DCS容灾演练。

Huawei Cloud DCS Master-Slave Failover (AZ Power-Off) Drill

Overview

This tool safely performs a DCS (Redis) master-slave failover drill on Huawei Cloud, verifying instance HA capability under AZ power-off (master node failure) scenarios. It runs a three-phase pipeline:

text

prepare (checks + risk assessment) -> execute (failover) -> report (generate report)

• prepare: query DCS instance, check IAM permissions, architecture hard-gate (ha + cross-AZ + RUNNING), generate experiment config directory, and run risk assessment
• execute: load config, safely execute master-slave failover (ChangeMasterStandby), poll for completion, data consistency check
• report: read config and execution result, generate comprehensive HTML report (instance info, safety checks, topology change, failover timeline, data consistency check)

Triggers / Trigger Conditions

Triggered when the user needs to perform a DCS master-slave failover, switchover, fault drill, AZ power-off drill, or failover experiment.

Triggers: DCS主备倒换, 主备倒换演练, DCS故障演练, 主备切换, 倒换实验, dcs failover, DCS failover, 高可用演练, 高可用验证, DCS HA切换, AZ断电演练, 主备互换, DCS switchover, Redis故障演练, DCS容灾演练

Trigger conditions:

• User mentions DCS master-slave failover or failover operation
• User needs to perform master-slave switchover on a ha-type DCS instance
• User needs a DCS fault drill or HA verification (AZ power-off scenario)
• User needs to swap master-slave or switchover on a DCS instance
• User needs to verify DCS HA / disaster recovery capability

Prerequisites

Condition	Description	Verification
hcloud CLI	Installed and authenticated (KooCLI >= 3.0.0)	hcloud version
AK/SK	Configured via hcloud profile. Script only checks if read-only calls succeed; actual auth is handled by hcloud profile	hcloud DCS ListInstances --cli-region=cn-south-1
Python 3.8+	To run scripts	python3 --version
Master-slave DCS instance	A ha-type instance exists in the target region	hcloud CLI
Instance status RUNNING	Instance must be running before failover	prepare phase auto-checks

Configuration

Shared config file config/config.json:

json

{
  "region": "cn-south-1",
  "instance_id": "<your-dcs-instance-id>"
}

Field	Required	Description
region	Yes	Huawei Cloud region ID (e.g. cn-south-1, cn-north-4)
instance_id	Yes	Target DCS instance ID

⚠️ You must modify config/config.json before running: replace region and instance_id with your own region and DCS instance ID. The default value is a placeholder; using it directly will fail.

ℹ️ No need to adjust hcloud env config: This skill does not require or recommend modifying hcloud profile (e.g. readTimeout). The script layer uses a sufficiently long subprocess timeout (120s) for CLI calls, and run_hcloud detects CLI timeout signatures (调用API超时 / context deadline exceeded / readTimeout), applying idempotent protection for synchronous mutation API timeouts (read-only recheck, no command replay). User hcloud env config can stay as-is.

Parameter Confirmation

Required Parameters

• region: Huawei Cloud region ID (e.g. cn-south-1, cn-north-4), configured in config/config.json
• instance_id: Target DCS master-slave instance ID, configured in config/config.json

Optional Parameters

• --phase: Pipeline phase to run (prepare / execute / report / all), default all
• --force: Skip critical risk blocking and force failover execution (requires manual confirmation)
• --config-dir: Config directory path, default experiments/

Conversational Usage (Recommended)

The user simply says "执行 DCS 主备倒换" (or an equivalent trigger), and the agent guides through the drill automatically. The user does not need to manually edit any config file.

Agent Guidance Flow

1. Probe instances: use hcloud DCS ListInstances --cli-region=<region> to query instances in the target region; if the user hasn't specified a region, probe by priority (cn-south-1 / cn-north-4 / cn-north-1 / cn-east-3) to find a drillable master-slave instance (ha + cross-AZ + RUNNING)
2. Confirm target: show candidate instances to the user, confirm the target (if only one candidate, can confirm directly)
3. Confirm execution: confirm user authorizes a real failover (failover is a mutation that swaps master-slave roles, must be confirmed). Options should only revolve around "execute/cancel", do not offer "prepare-only" options — the user's goal is to execute failover; prepare is just an internal pipeline phase
4. Write config: write the confirmed region / instance_id into config/config.json
5. Execute full pipeline: internally call python3 scripts/run.py to perform the real failover (if user confirms forced drill, internally append --force, no need to expose CLI args to user)
6. Present results: present CLI output as-is to the user, point out the report path

Questions for Missing Information

Missing info	Question
Target instance	"当前账号下 <区域> 有实例 X，确认用哪个？"
Region	"请提供目标区域（如 cn-south-1 华南-广州）"
Execution confirmation	"确认对实例 X 执行主备倒换？（会切换主备角色，引起短暂连接中断）"

⚠️ Questions must only revolve around "execute/cancel". Never offer "prepare-only" or "risk-assessment-only" options — those are invalid questions that treat internal pipeline phases as standalone goals.

Key Interaction Rules

• Failover is a mutation: must confirm with user before executing, never default-execute
• Risk disclosure: when prepare detects risks (including critical), must fully show each risk's impact and suggestion to the user
• Critical risk二次确认: when prepare detects critical risks, must show risks and confirm whether to proceed, never auto-proceed. Confirmation options are "强制演练 / 取消演练" (after user confirms forced drill, agent internally appends --force, not exposed to user), must not offer "prepare-only" invalid options
• DCS.4104 cooldown: if failover is rejected (instance in recovery), clearly tell the user "in failover cooldown, retry in a few minutes"
• Idempotent protection: if instance is already in target state (e.g. already failed over last round), inform user of current role status to avoid redundant failover

Quick Start

One-Click Full Pipeline

bash

# Full pipeline (skill invocation performs real failover)
python3 scripts/run.py

Phase-by-Phase Execution

bash

# 1. Prepare checks
python3 scripts/run.py --phase prepare

# 2. Execute failover
python3 scripts/run.py --phase execute
# If prepare detected critical risks and user confirmed to proceed, agent internally appends --force (not exposed to user):
#   python3 scripts/execute.py --config-dir experiments --force

# 3. Generate report
python3 scripts/run.py --phase report

Three Phases

Phase 1: prepare (Preparation Checks)

Query DCS resources and user permissions, generate config directory, and run architecture hard-gate + risk assessment.

• Query target DCS instance (ListInstances + ShowInstance to fill in cache_mode, etc.)
• Query master-slave node info (ShowNodesInformation)
• Check IAM permissions (hcloud auth readiness, DCS read/write permissions, failover permission)
• Generate config files (experiment.json, iam_policy.json, monitoring.json)
• Architecture hard-gate (11 checks): ha type, RUNNING, master-slave nodes exist, cross-AZ, etc.; required items block on failure
• Risk assessment (three-step pipeline):
  • Detect: read-only probe of instance status, master-slave node health, backup freshness, CES load metrics (CPU/memory/connections/latency)
  • Assess: grade each item by threshold (info/unknown/warning/critical), aggregate overall risk level (low/medium/blocked) with action recommendation
  • Block: results written to risk_assessment.json and embedded in readiness report; execute phase reads this file, blocks failover by default if critical items exist (user must confirm to proceed)

Output: config files and readiness report in experiments/ directory (including risk_assessment.json)

Phase 2: execute (Failover Execution)

⚠️ Mutation — master-slave failover actually swaps master-slave roles.

• Load config, pre-check (instance exists, ha, RUNNING, IAM permissions)
• Read risk_assessment.json and show risk assessment results; block failover if critical items exist and user hasn't confirmed
• Validate risk assessment consistency: block by default if risk_assessment.json region/instance_id doesn't match config.json (prevents "swap instance, use stale assessment to bypass")
• Real-time recheck before failover: immediately recheck instance status and node health before failover (prepare snapshot may be stale); block by default if critical anomaly found
• Call hcloud DCS ChangeMasterStandby to execute master-slave failover
• Poll node status every 5 seconds, wait for master-slave role swap to complete (timeout 120 seconds)
• Data consistency check (master node moved to slave AZ, node_id change, key count)
• Save execution result to execution_result.json

User interaction: failover is a real mutation; execute performs a real failover (no dry-run). Critical risks block by default; user must confirm to proceed. User-facing options are only "强制演练 / 取消演练" (internally via --force, not exposed to user).

Phase 3: report (Report Generation)

Read config and execution result, generate comprehensive HTML report.

Report includes: result banner, instance info, safety checks, master-slave topology change, failover timeline, data consistency check.

🔒 HTML encoding requirement (prevent stored XSS): all external data from cloud APIs (instance names, node info, check items, etc.) must be HTML-escaped via html.escape before insertion. Both scripts/report.py and scripts/prepare.py apply esc() / _esc() to all external data.

Output: templates/failover_report_YYYYMMDD_HHMMSS.html (standalone HTML, open in browser)

Risk Assessment

The prepare phase runs risk assessment in addition to architecture hard-gate checks. Assessment has three steps: detect -> assess -> block.

Detection Items (Read-Only Real-Time Probe)

Dimension	Data source (read-only API)	Metric
Instance health	DCS ShowInstance	Instance status RUNNING
Master-slave nodes	DCS ShowNodesInformation	Master/slave node existence and status
Backup safety net	DCS ListBackupRecords	Latest successful backup time, failed tasks in last 7 days
Load	CES ShowMetricData	CPU/memory/connections/latency average
Deployment	ShowInstance az_codes	Master-slave cross-AZ

Grading Rules (Threshold-Based)

Each detection item is mapped to one of four levels: info / unknown / warning / critical. Any critical item -> overall risk blocked; warning or unknown items -> medium; otherwise -> low. Threshold table and business semantics: see references/risk-model.md.

Blocking Mechanism

1. Console: prepare / execute phases prominently output overall level and each warning's impact and suggestion
2. Report: risk section embedded in readiness_report.html / readiness_report.md
3. Execution block: execute phase reads risk_assessment.json, blocks failover by default if critical items exist (unless user confirms to proceed)
4. Anti-mismatch: reject if risk assessment doesn't match the target instance; anti-stale-snapshot: real-time recheck of instance and node status before execution

⚠️ Missing data is never misjudged, but not auto-passed: when an API is unavailable (e.g. no permission, no CES data), the item is marked "unknown" with an explanation, never fabricating a critical risk; but unknown items raise overall risk to "medium", prompting manual confirmation before execution.

CES Monitoring Metrics

During prepare phase risk assessment, the last 1 hour of CES data is queried. The monitoring config defines the following 8 DCS metrics (namespace SYS.DCS, dimension dcs_instance_id):

Logic name	metric_name	Description	Unit
dcs_cpu_util	cpu_usage	CPU usage	%
dcs_mem_util	memory_usage	Memory usage	%
dcs_conn_usage	connections_usage	Connection usage	%
dcs_ops	instantaneous_ops	Instantaneous ops per second	count
dcs_connected_clients	connected_clients	Connected clients	count
dcs_command_max_rt	command_max_rt	Max command latency	us
dcs_in_flow	instantaneous_input_kbps	Instantaneous input flow	KiB/s
dcs_out_flow	instantaneous_output_kbps	Instantaneous output flow	KiB/s

Composite Scoring

Dimension	Weight	Basis
Failover	40%	Failover success / duration (>30s deducts 10 per 10s)
Data	30%	Consistency check pass ratio
Business	30%	Default 100 (reserved for CES data deduction)

Weighted total grade: >=85 excellent / >=70 good / >=60 fair / <60 poor

📌 Scoring is computed by the execute phase and written to execution_result.json (scores field); the HTML report does not currently render a scoring section.

IAM Permission Requirements

Permission	Description	Required
dcs:instance:list	List DCS instances	✅
dcs:instance:get	Get DCS instance detail	✅
dcs:instance:getNodes	Get master-slave node info	✅
dcs:instance:modify	Execute DCS master-slave failover (ChangeMasterStandby)	✅
dcs:backup:list	List backup records (risk assessment)	✅
ces:metricData:get	Query CES metric data	optional

📌 Full IAM policy template (with Resource URN scope) see references/iam-policies.md.

Directory Structure

text

huawei-cloud-dcs-failover/
├── SKILL.md                          # This file
├── config/
│   └── config.json                   # Shared config (region + instance_id)
├── scripts/
│   ├── config.py                     # Config loading + run_hcloud + shared utils
│   ├── prepare.py                    # Phase 1: prepare checks (architecture hard-gate + risk assessment)
│   ├── risk.py                       # Risk assessment core module
│   ├── execute.py                    # Phase 2: execute failover (risk blocking, user confirm to proceed)
│   ├── report.py                     # Phase 3: generate report
│   └── run.py                        # Unified entry point (one-click full pipeline)
├── references/
│   ├── cli-installation-guide.md     # hcloud CLI installation and auth guide
│   ├── iam-policies.md               # IAM policy description
│   ├── acceptance-criteria.md        # Acceptance criteria
│   ├── verification-method.md        # Verification method
│   └── risk-model.md                 # Risk threshold table and business semantics
├── experiments/                      # Intermediate artifacts (auto-cleaned before/after run)
│   ├── experiment.json
│   ├── iam_policy.json
│   ├── monitoring.json
│   ├── execution_result.json
│   ├── risk_assessment.json
│   ├── readiness_report.html
│   └── readiness_report.md
└── templates/                        # Final deliverable directory (HTML report)
    └── failover_report_YYYYMMDD_HHMMSS.html

Reference Documents

Document	Description
references/cli-installation-guide.md	hcloud CLI installation and auth config guide
references/iam-policies.md	IAM policy template and config description
references/acceptance-criteria.md	Functional acceptance criteria and checklist
references/verification-method.md	Environment and functional verification, end-to-end flow
references/risk-model.md	Risk assessment threshold table and business semantics (risk model)