Huawei Cloud Cce Cluster Upgrade Planner

Other

Use when planning CCE Kubernetes cluster version upgrades, evaluating upgrade path compatibility, addon compatibility, pre-upgrade difference checks, and estimating upgrade window duration to avoid insufficient upgrade windows. Covers version-specific breaking changes, deprecated APIs, 76-item pre-check checklist, in-place upgrade vs migration strategy comparison, and execution preview with two-step confirmation. Triggers: CCE 集群升级, 升级评估, 版本兼容, 升级窗口, 升级前检查, Kubernetes升级, addon升级, 差异检查, 升级方案, cluster upgrade, version upgrade, upgrade window, compatibility check

Install

openclaw skills install huawei-cloud-cce-cluster-upgrade-planner

CCE Cluster Upgrade Planner

Overview

Plan and assess CCE (Cloud Container Engine) Kubernetes cluster version upgrades using hcloud CLI. Covers upgrade path validation, 76-item pre-check checklist, addon compatibility, version-specific breaking changes, deprecated APIs, upgrade window estimation, and execution preview with two-step confirmation.

Architecture: hcloud CLI → CCE OpenAPI → Cluster Info / Upgrade Paths / Upgrade Workflow / Addon Info / Node Pool Info

Nodes

Standard workflow:

1. Collect cluster current state (version, nodes, addons, node pools)
2. Query upgrade paths via ListClusterUpgradePaths
3. Run pre-check via ShowClusterUpgradeInfo (or CreateUpgradeWorkFlow)
4. Evaluate addon compatibility for each installed addon
5. Estimate upgrade window based on node count, addon count, batch strategy
6. Generate upgrade plan with specific hcloud CLI commands (execution preview)
7. Provide rollback strategy and post-upgrade verification checklist

Prerequisites

Prerequisite check: hcloud (KooCLI) >= 7.2.2 required Run hcloud version to verify, and hcloud configure list to check profile exists.

hcloud version
hcloud configure list

Security Constraints

Dangerous Operation Confirmation Mechanism

This skill strictly enforces a two-step confirmation mechanism for upgrade execution preview.

All upgrade execution previews require explicit user confirmation before any changes are made. The process:

Step 1: Preview — Show upgrade commands, target version, affected components, and risk warnings

Step 2: Confirm & Execute — Only after user explicitly confirms

Operations Requiring Confirmation

Operation	Risk Level	Description
UpgradeCluster	🔴 Critical	Upgrades Kubernetes control plane, irreversible once started
UpgradeNodePool	🟠 High	Upgrades node pool Kubernetes version, nodes become temporarily unschedulable
CreateUpgradeWorkFlow	🟠 High	Creates upgrade workflow with pre-check, cluster upgrade, and post-check phases

Credential Security

Never expose AK/SK values in conversation, commands, or output
Never ask user to input AK/SK directly in conversation
Only use hcloud configure list to check credential status (presence only, not values)
Prefer profile mode or environment variables over explicit AK/SK parameters

Command Format Standard

CCE upgrade follows standard hcloud format:

hcloud CCE <Operation> --param=value --cli-region=<region> --cli-output=json

Upgrade-specific Parameter Rules

Cluster ID required: all upgrade operations need --cluster_id
Target version required: upgrade operations need --spec.clusterUpgradeAction.targetVersion=v1.XX
Addon upgrade uses array format: --spec.clusterUpgradeAction.addons.1.addonTemplateName=<name> --spec.clusterUpgradeAction.addons.1.version=<ver>
Node pool priority uses key-value format: --spec.clusterUpgradeAction.nodePoolOrder.key1=value1
Node selector uses nested format: --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.key=<label-key> --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.operator=In --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.value.1=<val>
Upgrade strategy: --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate (only in-place supported)
Batch size: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=<1-40> (default 20, recommended)
Batch scope: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster or NodePool

⚠️ Critical: Before constructing any upgrade command, always run hcloud CCE <Operation> --help to verify exact parameter names. CCE upgrade APIs have hundreds of parameters; the help output is the authoritative source.

Scenario Routing

User Intent	Reference Document
Full upgrade assessment (7-step workflow)	references/upgrade-workflow.md
Pre-upgrade checklist (76 items)	references/pre-upgrade-checklist.md
Addon compatibility matrix & upgrade order	references/addon-compatibility.md
K8s version upgrade path rules	references/k8s-version-matrix.md
Upgrade window time estimation	references/upgrade-window-estimation.md
Version-specific breaking changes	references/pre-upgrade-checklist.md
Deprecated API migration	references/pre-upgrade-checklist.md
Risk constraints & rollback	references/risk-rules.md
Output schema	references/output-schema.md

Core Commands

Step 1: Collect Cluster Current State

# Get cluster details (version, node count, addons)
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# List installed addons
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List node pools
hcloud CCE ListNodePools --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List nodes
hcloud CCE ListNodes --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 2: Query Upgrade Paths

# Get all available upgrade paths for the cluster
hcloud CCE ListClusterUpgradePaths --cli-region=<region> --cli-output=json

Step 3: Pre-Upgrade Check

# Get upgrade info (includes pre-check status)
hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Create upgrade workflow (triggers pre-check automatically)
hcloud CCE CreateUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --apiVersion=v3 \
  --kind=WorkFlowTask \
  --spec.targetVersion=<target-version> \
  --spec.clusterVersion=<current-version> \
  --cli-region=<region> \
  --cli-output=json

Step 4: Addon Compatibility Check

# Get addon details (check version compatibility)
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List addon template versions (check available versions for target K8s)
hcloud CCE ListAddonTemplates --cluster_id=<cluster-id> --addon_template_name=<addon-name> --cli-region=<region> --cli-output=json

Step 5: Upgrade Execution Preview

# Preview cluster upgrade (without confirm=true, no execution)
hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster \
  --cli-region=<region> \
  --cli-output=json

With addons (specify addon upgrade during cluster upgrade):

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.addons.1.addonTemplateName=coredns \
  --spec.clusterUpgradeAction.addons.1.operation=patch \
  --spec.clusterUpgradeAction.addons.1.version=<target-addon-version> \
  --cli-region=<region> \
  --cli-output=json

With node pool priority (control upgrade order):

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.nodePoolOrder.<nodepool-id>=<priority> \
  --cli-region=<region> \
  --cli-output=json

Step 6: Monitor Upgrade Progress

# Check upgrade workflow status
hcloud CCE ShowUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --cli-region=<region> \
  --cli-output=json

# List upgrade tasks
hcloud CCE ListUpgradeClusterTasks --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Pause upgrade (if issues found)
hcloud CCE PauseUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Continue paused upgrade
hcloud CCE ContinueUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Retry failed upgrade
hcloud CCE RetryUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 7: Cancel Upgrade Workflow

# Cancel upgrade workflow (status must not be Running/Success/Cancel)
hcloud CCE UpgradeWorkFlowUpdate \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --status.phase=Cancel \
  --cli-region=<region> \
  --cli-output=json

Upgrade Window Estimation Formula

See references/upgrade-window-estimation.md for detailed formula and examples.

Quick estimation:

T_total = T_control_plane + T_node_batch + T_addon + T_verify + T_buffer

T_control_plane = 10-30 minutes
T_node_batch = N_nodes / batch_size * (5-15 min/node)
T_addon = count(addons) * (5-15 min/addon)
T_verify = 30 minutes
T_buffer = 20% * (T_control_plane + T_node_batch + T_addon)

Example: 3-node-pool cluster (10 nodes, 5 addons) upgrading v1.23→v1.25:

Control plane: 15 min
Node batches: 10/4 = 3 batches (1+4+5 nodes), ~9 min each = 9 min total (first batch 1 node at 5 min, second 4 nodes at 8 min, third 5 nodes at 10 min)
Addon upgrade: 5 * 10 min = 50 min
Verification: 30 min
Buffer (20%): ~19 min
Total: ~123 min (2h 3min)

Upgrade Methods

Method	Advantages	Constraints	Recommended
In-place upgrade	Business pods not interrupted during control plane upgrade; nodes upgraded in batches; addons auto-upgraded	Nodes temporarily unschedulable during upgrade; Docker→Containerd runtime switch needed for v1.27+	Most scenarios (default)
Migration	Clean slate; no compatibility risk accumulation; skip multiple intermediate upgrades	Full workload redeployment; requires double resources; longer downtime	Cross-version jumps (e.g. v1.15→v1.28); incompatible runtime

Key Constraints

No skip-version upgrades: must follow upgrade path step by step (e.g., v1.21→v1.23→v1.25→v1.27→v1.28)
Patch version first: upgrade to latest patch before major version upgrade
Control plane before nodes: upgrade control plane first, then node pools
Addons after cluster: upgrade addons after cluster reaches target version
v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes, IP addresses change
Autoscaling paused: during control plane upgrade, autoscaling is paused; resumes after control plane done, but shrink waits until full completion

Parameter Reference

CCE Upgrade API Parameters

Parameter	Command	Required	Description	Constraints
`--cluster_id`	All upgrade operations	Yes	CCE cluster ID	Must reference existing cluster
`--spec.clusterUpgradeAction.targetVersion`	UpgradeCluster, CreateUpgradeWorkFlow	Yes	Target Kubernetes version	Must follow upgrade path rules (no skip-version)
`--spec.clusterUpgradeAction.strategy.type`	UpgradeCluster	Yes	Upgrade strategy type	Only `inPlaceRollingUpdate` supported
`--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep`	UpgradeCluster	No	Batch size per upgrade step	1-40, default 20
`--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope`	UpgradeCluster	No	Batch upgrade scope	`Cluster` or `NodePool`
`--spec.clusterUpgradeAction.addons.N.addonTemplateName`	UpgradeCluster	No	Addon template name	Must match installed addon name
`--spec.clusterUpgradeAction.addons.N.version`	UpgradeCluster	No	Target addon version	Must be compatible with target K8s version
`--spec.clusterUpgradeAction.nodePoolOrder.key`	UpgradeCluster	No	Node pool upgrade priority	key=nodepool-id, value=priority number
`--metadata.apiVersion`	UpgradeCluster	Yes	API version	`v3`
`--metadata.kind`	UpgradeCluster	Yes	Resource kind	`UpgradeTask`
`--cli-region`	All hcloud commands	Required	Region ID	Config value or env variable
`--cli-output`	All hcloud commands	Recommended	Output format	`json` recommended

Upgrade Workflow Parameters

Parameter	Command	Required	Description	Constraints
`--spec.targetVersion`	CreateUpgradeWorkFlow	Yes	Target version for pre-check	Must follow upgrade path
`--spec.clusterVersion`	CreateUpgradeWorkFlow	Yes	Current cluster version	Must match actual cluster version
`--upgrade_workflow_id`	ShowUpgradeWorkFlow, UpgradeWorkFlowUpdate	Yes	Workflow task ID	Obtained from CreateUpgradeWorkFlow response

Output Format

Upgrade Assessment Report

{
  "cluster_id": "abc123",
  "current_version": "v1.23",
  "target_version": "v1.25",
  "upgrade_paths": ["v1.23 → v1.24 → v1.25"],
  "pre_check_status": "passed|failed|warning",
  "pre_check_items": { "total": 76, "passed": 72, "failed": 4 },
  "addon_compatibility": [
    { "addon": "coredns", "current": "1.23", "compatible": true }
  ],
  "estimated_window_minutes": 123,
  "risk_level": "low|medium|high",
  "recommended_strategy": "inPlaceRollingUpdate",
  "execution_commands": ["hcloud CCE UpgradeCluster ..."]
}

Upgrade Execution Preview

{
  "cluster_id": "abc123",
  "step": "cluster_upgrade_preview",
  "target_version": "v1.25",
  "strategy": "inPlaceRollingUpdate",
  "batch_size": 20,
  "addons_upgrade": ["coredns → 1.25"],
  "estimated_time": "2h 3min",
  "confirmation_required": true,
  "warning": "Control plane upgrade is irreversible once started"
}

Verification

Pre-Upgrade Verification

hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Verify preCheckStatus is passed
Check all 76 items in preCheckItems

Post-Upgrade Verification

# Check cluster version
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# Check addon status
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Check node status
kubectl get nodes

# Check workload health
kubectl get pods -A

Notes

No skip-version upgrades: must follow upgrade path step by step
In-place upgrade is the only supported strategy: inPlaceRollingUpdate only
v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes with different IP addresses
Autoscaling paused during upgrade: cluster autoscaling pauses during control plane upgrade
Addon upgrade must follow cluster upgrade: addons upgrade after cluster reaches target version
Pre-check is mandatory: never skip ShowClusterUpgradeInfo or CreateUpgradeWorkFlow pre-check

Best Practices

Always run pre-check first — use CreateUpgradeWorkFlow or ShowClusterUpgradeInfo to validate readiness
Upgrade patch version first — ensure current patch is latest before major version upgrade
Business low-traffic window — schedule upgrades during low-traffic periods
Keep ≥2 nodes per node pool for redundancy during upgrade
Monitor upgrade progress — check ShowUpgradeWorkFlow status after each phase
Verify after each phase — run post-upgrade verification after control plane and node upgrades
Rollback plan — use backup data and PauseUpgradeClusterTask if issues arise

References

Document	Description
upgrade-workflow.md	Full 7-step upgrade workflow detail
pre-upgrade-checklist.md	76-item pre-check checklist and version-specific breaking changes
addon-compatibility.md	Addon compatibility matrix, DaemonSet plugins, upgrade order
k8s-version-matrix.md	Official CCE upgrade path table and patch version rules
upgrade-window-estimation.md	Upgrade window estimation formula, batch strategy, examples
risk-rules.md	Risk constraints, rollback strategies, guardrails
output-schema.md	Assessment report and execution preview JSON schema