Huawei Cloud Cce Cluster Upgrade Planner

Use when planning CCE Kubernetes cluster version upgrades, evaluating upgrade path compatibility, addon compatibility, pre-upgrade difference checks, and estimating upgrade window duration to avoid insufficient upgrade windows. Covers version-specific breaking changes, deprecated APIs, 76-item pre-check checklist, in-place upgrade vs migration strategy comparison, and execution preview with two-step confirmation. Triggers: CCE 集群升级, 升级评估, 版本兼容, 升级窗口, 升级前检查, Kubernetes升级, addon升级, 差异检查, 升级方案, cluster upgrade, version upgrade, upgrade window, compatibility check

CCE Cluster Upgrade Planner

Overview

Plan and assess CCE (Cloud Container Engine) Kubernetes cluster version upgrades using hcloud CLI. Covers upgrade path validation, 76-item pre-check checklist, addon compatibility, version-specific breaking changes, deprecated APIs, upgrade window estimation, and execution preview with two-step confirmation.

Architecture: hcloud CLI → CCE OpenAPI → Cluster Info / Upgrade Paths / Upgrade Workflow / Addon Info / Node Pool Info

Nodes

Standard workflow:

text

1. Collect cluster current state (version, nodes, addons, node pools)
2. Query upgrade paths via ListClusterUpgradePaths
3. Run pre-check via ShowClusterUpgradeInfo (or CreateUpgradeWorkFlow)
4. Evaluate addon compatibility for each installed addon
5. Estimate upgrade window based on node count, addon count, batch strategy
6. Generate upgrade plan with specific hcloud CLI commands (execution preview)
7. Provide rollback strategy and post-upgrade verification checklist

Prerequisites

Prerequisite check: hcloud (KooCLI) >= 7.2.2 required Run hcloud version to verify, and hcloud configure list to check profile exists.

bash

hcloud version
hcloud configure list

Security Constraints

Dangerous Operation Confirmation Mechanism

This skill strictly enforces a two-step confirmation mechanism for upgrade execution preview.

All upgrade execution previews require explicit user confirmation before any changes are made. The process:

Step 1: Preview — Show upgrade commands, target version, affected components, and risk warnings

Step 2: Confirm & Execute — Only after user explicitly confirms

Operations Requiring Confirmation

Operation	Risk Level	Description
UpgradeCluster	🔴 Critical	Upgrades Kubernetes control plane, irreversible once started
UpgradeNodePool	🟠 High	Upgrades node pool Kubernetes version, nodes become temporarily unschedulable
CreateUpgradeWorkFlow	🟠 High	Creates upgrade workflow with pre-check, cluster upgrade, and post-check phases

Credential Security

• Never expose AK/SK values in conversation, commands, or output
• Never ask user to input AK/SK directly in conversation
• Only use hcloud configure list to check credential status (presence only, not values)
• Prefer profile mode or environment variables over explicit AK/SK parameters

Command Format Standard

CCE upgrade follows standard hcloud format:

bash

hcloud CCE <Operation> --param=value --cli-region=<region> --cli-output=json

Upgrade-specific Parameter Rules

1. Cluster ID required: all upgrade operations need --cluster_id
2. Target version required: upgrade operations need --spec.clusterUpgradeAction.targetVersion=v1.XX
3. Addon upgrade uses array format: --spec.clusterUpgradeAction.addons.1.addonTemplateName=<name> --spec.clusterUpgradeAction.addons.1.version=<ver>
4. Node pool priority uses key-value format: --spec.clusterUpgradeAction.nodePoolOrder.key1=value1
5. Node selector uses nested format: --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.key=<label-key> --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.operator=In --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.value.1=<val>
6. Upgrade strategy: --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate (only in-place supported)
7. Batch size: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=<1-40> (default 20, recommended)
8. Batch scope: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster or NodePool

⚠️ Critical: Before constructing any upgrade command, always run hcloud CCE <Operation> --help to verify exact parameter names. CCE upgrade APIs have hundreds of parameters; the help output is the authoritative source.

Scenario Routing

User Intent	Reference Document
Full upgrade assessment (7-step workflow)	references/upgrade-workflow.md
Pre-upgrade checklist (76 items)	references/pre-upgrade-checklist.md
Addon compatibility matrix & upgrade order	references/addon-compatibility.md
K8s version upgrade path rules	references/k8s-version-matrix.md
Upgrade window time estimation	references/upgrade-window-estimation.md
Version-specific breaking changes	references/pre-upgrade-checklist.md
Deprecated API migration	references/pre-upgrade-checklist.md
Risk constraints & rollback	references/risk-rules.md
Output schema	references/output-schema.md

Core Commands

Step 1: Collect Cluster Current State

bash

# Get cluster details (version, node count, addons)
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# List installed addons
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List node pools
hcloud CCE ListNodePools --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List nodes
hcloud CCE ListNodes --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 2: Query Upgrade Paths

bash

# Get all available upgrade paths for the cluster
hcloud CCE ListClusterUpgradePaths --cli-region=<region> --cli-output=json

Step 3: Pre-Upgrade Check

bash

# Get upgrade info (includes pre-check status)
hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Create upgrade workflow (triggers pre-check automatically)
hcloud CCE CreateUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --apiVersion=v3 \
  --kind=WorkFlowTask \
  --spec.targetVersion=<target-version> \
  --spec.clusterVersion=<current-version> \
  --cli-region=<region> \
  --cli-output=json

Step 4: Addon Compatibility Check

bash

# Get addon details (check version compatibility)
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List addon template versions (check available versions for target K8s)
hcloud CCE ListAddonTemplates --cluster_id=<cluster-id> --addon_template_name=<addon-name> --cli-region=<region> --cli-output=json

Step 5: Upgrade Execution Preview

bash

# Preview cluster upgrade (without confirm=true, no execution)
hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster \
  --cli-region=<region> \
  --cli-output=json

With addons (specify addon upgrade during cluster upgrade):

bash

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.addons.1.addonTemplateName=coredns \
  --spec.clusterUpgradeAction.addons.1.operation=patch \
  --spec.clusterUpgradeAction.addons.1.version=<target-addon-version> \
  --cli-region=<region> \
  --cli-output=json

With node pool priority (control upgrade order):

bash

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.nodePoolOrder.<nodepool-id>=<priority> \
  --cli-region=<region> \
  --cli-output=json

Step 6: Monitor Upgrade Progress

bash

# Check upgrade workflow status
hcloud CCE ShowUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --cli-region=<region> \
  --cli-output=json

# List upgrade tasks
hcloud CCE ListUpgradeClusterTasks --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Pause upgrade (if issues found)
hcloud CCE PauseUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Continue paused upgrade
hcloud CCE ContinueUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Retry failed upgrade
hcloud CCE RetryUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 7: Cancel Upgrade Workflow

bash

# Cancel upgrade workflow (status must not be Running/Success/Cancel)
hcloud CCE UpgradeWorkFlowUpdate \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --status.phase=Cancel \
  --cli-region=<region> \
  --cli-output=json

Upgrade Window Estimation Formula

See references/upgrade-window-estimation.md for detailed formula and examples.

Quick estimation:

text

T_total = T_control_plane + T_node_batch + T_addon + T_verify + T_buffer

T_control_plane = 10-30 minutes
T_node_batch = N_nodes / batch_size * (5-15 min/node)
T_addon = count(addons) * (5-15 min/addon)
T_verify = 30 minutes
T_buffer = 20% * (T_control_plane + T_node_batch + T_addon)

Example: 3-node-pool cluster (10 nodes, 5 addons) upgrading v1.23→v1.25:

• Control plane: 15 min
• Node batches: 10/4 = 3 batches (1+4+5 nodes), ~9 min each = 9 min total (first batch 1 node at 5 min, second 4 nodes at 8 min, third 5 nodes at 10 min)
• Addon upgrade: 5 * 10 min = 50 min
• Verification: 30 min
• Buffer (20%): ~19 min
• Total: ~123 min (2h 3min)

Upgrade Methods

Method	Advantages	Constraints	Recommended
In-place upgrade	Business pods not interrupted during control plane upgrade; nodes upgraded in batches; addons auto-upgraded	Nodes temporarily unschedulable during upgrade; Docker→Containerd runtime switch needed for v1.27+	Most scenarios (default)
Migration	Clean slate; no compatibility risk accumulation; skip multiple intermediate upgrades	Full workload redeployment; requires double resources; longer downtime	Cross-version jumps (e.g. v1.15→v1.28); incompatible runtime

Key Constraints

• No skip-version upgrades: must follow upgrade path step by step (e.g., v1.21→v1.23→v1.25→v1.27→v1.28)
• Patch version first: upgrade to latest patch before major version upgrade
• Control plane before nodes: upgrade control plane first, then node pools
• Addons after cluster: upgrade addons after cluster reaches target version
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes, IP addresses change
• Autoscaling paused: during control plane upgrade, autoscaling is paused; resumes after control plane done, but shrink waits until full completion

Parameter Reference

CCE Upgrade API Parameters

Parameter	Command	Required	Description	Constraints
--cluster_id	All upgrade operations	Yes	CCE cluster ID	Must reference existing cluster
--spec.clusterUpgradeAction.targetVersion	UpgradeCluster, CreateUpgradeWorkFlow	Yes	Target Kubernetes version	Must follow upgrade path rules (no skip-version)
--spec.clusterUpgradeAction.strategy.type	UpgradeCluster	Yes	Upgrade strategy type	Only inPlaceRollingUpdate supported
--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep	UpgradeCluster	No	Batch size per upgrade step	1-40, default 20
--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope	UpgradeCluster	No	Batch upgrade scope	Cluster or NodePool
--spec.clusterUpgradeAction.addons.N.addonTemplateName	UpgradeCluster	No	Addon template name	Must match installed addon name
--spec.clusterUpgradeAction.addons.N.version	UpgradeCluster	No	Target addon version	Must be compatible with target K8s version
--spec.clusterUpgradeAction.nodePoolOrder.key	UpgradeCluster	No	Node pool upgrade priority	key=nodepool-id, value=priority number
--metadata.apiVersion	UpgradeCluster	Yes	API version	v3
--metadata.kind	UpgradeCluster	Yes	Resource kind	UpgradeTask
--cli-region	All hcloud commands	Required	Region ID	Config value or env variable
--cli-output	All hcloud commands	Recommended	Output format	json recommended

Upgrade Workflow Parameters

Parameter	Command	Required	Description	Constraints
--spec.targetVersion	CreateUpgradeWorkFlow	Yes	Target version for pre-check	Must follow upgrade path
--spec.clusterVersion	CreateUpgradeWorkFlow	Yes	Current cluster version	Must match actual cluster version
--upgrade_workflow_id	ShowUpgradeWorkFlow, UpgradeWorkFlowUpdate	Yes	Workflow task ID	Obtained from CreateUpgradeWorkFlow response

Output Format

Upgrade Assessment Report

json

{
  "cluster_id": "abc123",
  "current_version": "v1.23",
  "target_version": "v1.25",
  "upgrade_paths": ["v1.23 → v1.24 → v1.25"],
  "pre_check_status": "passed|failed|warning",
  "pre_check_items": { "total": 76, "passed": 72, "failed": 4 },
  "addon_compatibility": [
    { "addon": "coredns", "current": "1.23", "compatible": true }
  ],
  "estimated_window_minutes": 123,
  "risk_level": "low|medium|high",
  "recommended_strategy": "inPlaceRollingUpdate",
  "execution_commands": ["hcloud CCE UpgradeCluster ..."]
}

Upgrade Execution Preview

json

{
  "cluster_id": "abc123",
  "step": "cluster_upgrade_preview",
  "target_version": "v1.25",
  "strategy": "inPlaceRollingUpdate",
  "batch_size": 20,
  "addons_upgrade": ["coredns → 1.25"],
  "estimated_time": "2h 3min",
  "confirmation_required": true,
  "warning": "Control plane upgrade is irreversible once started"
}

Verification

Pre-Upgrade Verification

bash

hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

• Verify preCheckStatus is passed
• Check all 76 items in preCheckItems

Post-Upgrade Verification

bash

# Check cluster version
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# Check addon status
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Check node status
kubectl get nodes

# Check workload health
kubectl get pods -A

Notes

• No skip-version upgrades: must follow upgrade path step by step
• In-place upgrade is the only supported strategy: inPlaceRollingUpdate only
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes with different IP addresses
• Autoscaling paused during upgrade: cluster autoscaling pauses during control plane upgrade
• Addon upgrade must follow cluster upgrade: addons upgrade after cluster reaches target version
• Pre-check is mandatory: never skip ShowClusterUpgradeInfo or CreateUpgradeWorkFlow pre-check

Best Practices

1. Always run pre-check first — use CreateUpgradeWorkFlow or ShowClusterUpgradeInfo to validate readiness
2. Upgrade patch version first — ensure current patch is latest before major version upgrade
3. Business low-traffic window — schedule upgrades during low-traffic periods
4. Keep ≥2 nodes per node pool for redundancy during upgrade
5. Monitor upgrade progress — check ShowUpgradeWorkFlow status after each phase
6. Verify after each phase — run post-upgrade verification after control plane and node upgrades
7. Rollback plan — use backup data and PauseUpgradeClusterTask if issues arise

References

Document	Description
upgrade-workflow.md	Full 7-step upgrade workflow detail
pre-upgrade-checklist.md	76-item pre-check checklist and version-specific breaking changes
addon-compatibility.md	Addon compatibility matrix, DaemonSet plugins, upgrade order
k8s-version-matrix.md	Official CCE upgrade path table and patch version rules
upgrade-window-estimation.md	Upgrade window estimation formula, batch strategy, examples
risk-rules.md	Risk constraints, rollback strategies, guardrails
output-schema.md	Assessment report and execution preview JSON schema