Huawei Cloud Cce Storage Failure Diagnoser

Huawei Cloud CCE Storage failure diagnosis skill using Python SDK dispatcher. Use this skill when the user wants to: (1) diagnose PVC Pending, volume mount failures, (2) analyze EVS disk issues, (3) diagnose storage class and CSI driver errors, (4) check PV/PVC binding status and storage capacity. Trigger: user mentions "storage failure", "存储故障", "PVC Pending", "PVC 挂载失败", "volume mount error", "卷挂载错误", "EVS disk", "云硬盘", "PV failure", "PV 异常", "CSI driver error", "CSI 驱动异常", "存储诊断", "FailedMount", "FailedAttachVolume"

Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, kubectl, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, kubectl, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview

This skill diagnoses CCE/Kubernetes storage failures across PVC provisioning, scheduling/binding, attach/mount, runtime I/O, capacity, permission, and teardown stages. It uses the local Python dispatcher (scripts/huawei-cloud.py) to call the Huawei Cloud Python SDK and Kubernetes client APIs, collecting PVC/PV/StorageClass/Pod/Node/Event/VolumeAttachment evidence, Everest CSI logs, Kubelet /stats/summary, and cloud-side storage information. It produces a complete Markdown diagnosis report with process, evidence, conclusion, confidence, and remediation guidance.

Related Skills

Skill	Purpose
huawei-cloud-cce-node-failure-diagnoser	Node-level failure diagnosis (scheduling, node resource issues)
huawei-cloud-cce-network-failure-diagnoser	Network failure diagnosis (Service/security group/ACL chain)
huawei-cloud-cce-pod-failure-diagnoser	Pod-level failure diagnosis
huawei-cloud-cce-auto-remediation-runner	Execute remediation actions (delete residual Pods, migrate workloads, expand storage, fix cloud resources)
huawei-cloud-cce-metric-analyzer	Metric trend analysis
huawei-cloud-cce-observability-context-builder	Observability context enrichment

Capabilities

1. One-shot storage failure diagnosis with structured evidence and Markdown report (huawei_storage_failure_diagnose)
2. PVC/PV/StorageClass/VolumeAttachment collection (huawei_get_cce_pvcs, huawei_get_cce_pvs, huawei_get_cce_storageclasses, huawei_get_cce_volumeattachments)
3. Node Kubelet /stats/summary proxy-read for capacity and inode analysis (huawei_get_cce_node_stats_summary)
4. Everest CSI driver/controller log retrieval with auto-sanitization (huawei_get_cce_everest_csi_logs)
5. Cloud-side EVS/SFS/SFS Turbo supplementary evidence (huawei_list_evs, huawei_get_evs_metrics, huawei_list_sfs, huawei_list_sfs_turbo)
6. Network supplementary evidence for SFS/NFS (huawei_list_security_groups, huawei_list_vpc_acls)
7. Pod, Node, and Event Kubernetes evidence (huawei_get_cce_pods, huawei_get_kubernetes_nodes, huawei_get_cce_events)

Typical Use Cases

• Diagnose a PVC stuck in Pending state
• Investigate Pod stuck in ContainerCreating with FailedMount or FailedAttachVolume events
• Analyze EVS disk attach failures, residual attachment locks, or per-node disk count limits
• Troubleshoot SFS/SFS Turbo NFS mount timeouts or network data-plane blocking
• Resolve OBS bucket access 403 errors, IAM delegation or AK/SK credential failures
• Diagnose runtime read-only filesystem, capacity or inode exhaustion
• Investigate ConfigMap/Secret subPath mount deadlocks
• Resolve PVC stuck in Terminating due to protection finalizers
• Check StorageClass provisioning or CSI driver errors

What This Skill Does NOT Handle

1. Creating, modifying, or deleting PVC/PV/Pod resources
2. Removing finalizers or force-detaching EVS disks
3. Modifying StorageClass, IAM delegations, AK/SK Secrets, security groups, or ACLs
4. Executing kubectl exec, node SSH, packet capture, stress tests, or fsck
5. Any write operations on the data plane or control plane

---

Prerequisites

Python Dependencies

The dispatcher script requires Python >= 3.6 and the following packages:

• huaweicloudsdkcore
• huaweicloudsdkcce
• huaweicloudsdkevs
• huaweicloudsdksfs
• huaweicloudsdkvpc
• huaweicloudsdkiam
• huaweicloudsdkces
• kubernetes

Run environment check before first use (see Verification section). The venv is auto-created by check_env; on Linux/macOS use .venv/bin/python3, on Windows use .venv/Scripts/python3.exe.

Credential Configuration

Variable	Required	Description
HW_ACCESS_KEY	Yes	Huawei Cloud Access Key
HW_SECRET_KEY	Yes	Huawei Cloud Secret Key
HW_REGION_NAME	No	Default region (overrides region param if set); default cn-north-4
HW_PROJECT_ID	No	Project ID (auto-obtained via IAM API when not set)
HW_SECURITY_TOKEN	No	Required when using temporary AK/SK

Security constraints:

1. Never persist AK/SK/Token/Certificate to disk or long-term memory
2. AK/SK exists only in the current request call stack and is released on completion
3. Only non-sensitive project IDs may be cached in process memory (never written to disk)
4. All temporary certificate files must be deleted immediately after use
5. Never leak AK/SK or other sensitive information in logs, responses, or errors
6. Never send authentication information to any third-party server

Do not output the values of the above environment variables.

IAM Permissions

This skill requires read-only IAM permissions for CCE, EVS, SFS, OBS, VPC, and CES services. Minimum required permissions:

Service	Permission	Purpose
CCE	cce:cluster:get, cce:node:get	Read cluster and node info
CCE	cce:pod:get, cce:pvc:get	Read Pod and PVC status
EVS	evs:disk:list, evs:disk:get	Read EVS disk details
EVS	evs:cloudvolume:list	List cloud volumes
VPC	vpc:securityGroup:get, vpc:firewall:get	Read security groups and ACLs
SFS	sfs:share:get, sfs:share:list	Read SFS/SFS Turbo shares

If a permission check fails, verify AK/SK configuration, confirm the user has the required read-only permissions, and check that the IAM policy is active (policies typically take effect within 5-10 minutes).

---

Core Tools

All actions are invoked via the dispatcher script:

bash

python3 scripts/huawei-cloud.py <action> region=<region> cluster_id=<cluster_id> [key=value ...]

Primary Diagnosis Action

bash

python3 scripts/huawei-cloud.py huawei_storage_failure_diagnose \
  region=cn-north-4 cluster_id=<cluster_id> \
  namespace=default pvc_name=<pvc_name> \
  include_stats=true include_logs=true include_cloud=false

Returns structured evidence + report_markdown (complete Markdown diagnosis report).

Recommended defaults: include_stats=true, include_logs=true, include_cloud=false. Set include_cloud=true when you need EVS/SFS/SFS Turbo and security group/ACL supplementary evidence.

Kubernetes Evidence Actions

Action	Required Params	Optional Params	Description
huawei_get_cce_pvcs	region, cluster_id	namespace, pvc_name	List PVCs
huawei_get_cce_pvs	region, cluster_id	pv_name	List PVs
huawei_get_cce_storageclasses	region, cluster_id	-	List StorageClasses with provisioner, parameters, volumeBindingMode
huawei_get_cce_volumeattachments	region, cluster_id	-	List VolumeAttachments with attached status, attachError, detachError
huawei_get_cce_node_stats_summary	region, cluster_id	-	Proxy-read node /stats/summary; parse PVC usedBytes/capacityBytes and inode
huawei_get_cce_everest_csi_logs	region, cluster_id	-	Read Everest CSI driver/controller logs (auto-sanitized)
huawei_get_cce_events	region, cluster_id	-	List cluster events
huawei_get_cce_pods	region, cluster_id	namespace, pod_name	List Pods
huawei_get_kubernetes_nodes	region, cluster_id	-	List Kubernetes nodes with labels, taints, conditions

Cloud Supplementary Evidence Actions

Action	Required Params	Optional Params	Description
huawei_list_evs	region	disk_id, availability_zone	List EVS disks
huawei_get_evs_metrics	region, disk_id	-	Get EVS disk I/O metrics
huawei_list_sfs	region	-	List SFS file systems
huawei_list_sfs_turbo	region	-	List SFS Turbo file systems
huawei_list_security_groups	region	-	List VPC security groups (for SFS/NFS network analysis)
huawei_list_vpc_acls	region	-	List VPC network ACLs (for SFS/NFS network analysis)

---

Parameter Reference

huawei_storage_failure_diagnose

Parameter	Required	Default	Description
region	Yes	-	Huawei Cloud region (e.g., cn-north-4)
cluster_id	Yes	-	CCE cluster ID
namespace	No	-	Kubernetes namespace (recommended for PVC Pending/Terminating/capacity issues)
pvc_name	No	-	Specific PVC name
pod_name	No	-	Specific Pod name (recommended for Pod Pending/ContainerCreating/IO anomalies)
failure_symptom	No	-	Symptom description, e.g., "PVC Pending", "FailedMount mount.nfs timeout", "OBS 403", "Read-only file system", "PVC Terminating"
include_stats	No	true	Include node /stats/summary for capacity/inode analysis
include_logs	No	true	Include Everest CSI driver/controller logs
include_cloud	No	false	Include EVS/SFS/SFS Turbo and security group/ACL cloud-side evidence

Common Parameters (All Actions)

Parameter	Required	Default	Description
region	Yes	-	Huawei Cloud region
cluster_id	Yes*	-	CCE cluster ID (required for CCE/K8s actions; not required for pure cloud actions)
ak	No	env HW_ACCESS_KEY	Huawei Cloud AK
sk	No	env HW_SECRET_KEY	Huawei Cloud SK
project_id	No	auto-obtained	Project ID (auto-obtained via IAM API when not set)

*Required for CCE/Kubernetes actions. Not required for pure cloud-side actions like huawei_list_evs, huawei_list_security_groups.

---

Output Format

The primary action huawei_storage_failure_diagnose returns structured JSON with an embedded report_markdown. See references/output-schema.md for the full JSON response schema.

json

{
  "success": true,
  "action": "huawei_storage_failure_diagnose",
  "region": "cn-north-4",
  "cluster_id": "cluster-id",
  "namespace": "default",
  "conclusion": "high signal conclusion",
  "confidence": "High",
  "findings": [
    {
      "stage": "Mount stage failure",
      "type": "EVSNodeAttachLimitExceeded",
      "title": "VolumeAttachment attached=false; error indicates ECS per-node disk count limit reached",
      "confidence": 0.94,
      "severity": "critical",
      "evidence": [],
      "recommendation": []
    }
  ],
  "top_causes": [],
  "snapshot": {},
  "report_markdown": "# CCE Storage Failure Automated Diagnosis Report\n..."
}

Required Markdown Report Sections

When report_markdown is present, use it as the final report body. You may add clarifications the user requests, but do not discard evidence tables.

The report_markdown must contain these headings:

• # CCE Storage Failure Automated Diagnosis Report
• ## 1. Diagnosis Overview
• ## 2. Investigation Process
• ## 3. Key Object Relationships
• ## 4. Evidence Matrix
• ## 5. Diagnosis Conclusion
• ## 6. Recommended Actions and Verification Standards
• ## 7. Data Gaps and Manual Confirmation

Finding Types

Common type values in findings:

Type	Description
NormalWaitForFirstConsumer	PVC Pending with WaitForFirstConsumer; normal behavior awaiting Pod scheduling
EVSQuotaExceeded	EVS cloud disk quota exceeded
SFSSubnetIPInsufficient	SFS/SFS Turbo subnet available IP or mount target allocation failure
OBSBucketNameInvalid	OBS bucket name conflict or invalid naming
EVSAvailabilityZoneSchedulingConflict	EVS single-AZ affinity prevents Pod scheduling to storage AZ
LocalPVNodeOffline	Local PV host node down/offline
VolumeAttachmentNotCreated	K8s control plane has not issued attach instruction
EVSNodeAttachLimitExceeded	ECS per-node attached disk count limit reached
EVSResidualAttachmentLock	EVS residual node occupancy or underlying lock not released
EVSAttachFailed	EVS attach failure (general)
HostKernelMountFailed	Cloud-side attached but host kernel/filesystem mount failed
SFSNfsNetworkBlocked	SFS/SFS Turbo NFS mount timeout due to network data-plane blocking
OBSCredentialInvalid	OBS IAM delegation changed, AK/SK Secret invalid, or bucket permission error
StoragePermissionDenied	Permission denied / forbidden / access denied (general)
PVCCapacityExhausted	PVC capacity usage > 95%
PVCInodeExhausted	PVC inode usage > 95%
ReadOnlyFilesystemProtection	Linux read-only filesystem protection triggered
ConfigMapSecretSubPathDeadlock	ConfigMap/Secret subPath mount point deadlock
PVCProtectionBlocked	PVC Terminating with kubernetes.io/pvc-protection finalizer
StorageIOError	Runtime storage I/O errors

---

Verification

Environment Check

Before first use, run the environment check script to install dependencies and validate credentials:

• Linux/macOS: skill action=exec: bash skill://scripts/check_env.sh
• Windows: skill action=exec: powershell -ExecutionPolicy Bypass -File skill://scripts/check_env.ps1

The script checks: Python >= 3.6, install dependencies, validate SDK, validate credentials, validate service availability.

Diagnosis Verification

1. Run environment check and confirm all checks pass
2. Execute huawei_storage_failure_diagnose with a known region and cluster_id:

   bash

   python3 scripts/huawei-cloud.py huawei_storage_failure_diagnose \
     region=cn-north-4 cluster_id=<cluster_id> include_stats=true include_logs=true
   

3. Verify the returned JSON contains success=true, findings array, and report_markdown
4. Check that the Markdown report contains all required sections (see references/output-schema.md)
5. Compare diagnosis conclusions against known failure patterns

---

Best Practices

1. Always call huawei_storage_failure_diagnose first; use individual tools only as fallback or for raw evidence
2. Provide namespace and pvc_name/pod_name when possible to narrow diagnosis scope
3. Set include_cloud=true only when you need cloud-side (EVS/SFS/OBS) supplementary evidence
4. For NFS/SFS mount timeouts, always supplement with security group and VPC ACL checks
5. For OBS 403 errors, focus on Everest CSI logs and event messages rather than cloud-side queries
6. Conclusion confidence is ranked by evidence strength, not by stage priority
7. Never write guesses as conclusions; output evidence gaps explicitly
8. For any remediation actions, only output proposed plan and verification standards, then hand off to huawei-cloud-cce-auto-remediation-runner for user confirmation

---

Reference Documents

Document	Description
references/workflow.md	Diagnosis triage flow, reusable capabilities, and stage-by-stage pipeline
references/output-schema.md	Output JSON schema and required Markdown report sections
references/risk-rules.md	Risk boundary rules: allowed read actions, prohibited write actions, and high-risk handoff
Huawei Cloud Python SDK Documentation	SDK reference
Huawei Cloud API Explorer	API interactive explorer

---

Notes

1. This skill is read-only diagnosis only — it never deletes PVC/PV/Pod, patches finalizers, force-detaches/attaches EVS, or modifies any StorageClass/IAM/Secret/SecurityGroup/ACL
2. Never expose or log AK/SK or environment variable values
3. All actions are executed via python3 scripts/huawei-cloud.py <action>; do not use hcloud CLI, kubectl, or direct API calls
4. PVC Terminating: never directly suggest removing kubernetes.io/pvc-protection finalizer; must first prove no Pod references and no business data risk
5. EVS residual mount or read-only filesystem scenarios: never suggest force-unmount, force-attach, or direct restart of database-class workloads before confirming filesystem consistency
6. ConfigMap/Secret resourceVersion has no natural update timestamp; use managedFields.time, Pod timestamps, and FailedMount events as circumstantial evidence only
7. Cross-diagnosis handoff: scheduling/node resource issues -> huawei-cloud-cce-node-failure-diagnoser; Service/security group/ACL chain -> huawei-cloud-cce-network-failure-diagnoser; remediation actions -> huawei-cloud-cce-auto-remediation-runner

---

Common Pitfalls

Pitfall	Correct Approach
Treating WaitForFirstConsumer PVC Pending as a failure	A PVC in Pending state with WaitForFirstConsumer volumeBindingMode and no associated Pod is normal behavior, not a failure
Diagnosing scheduling failures without AZ context	EVS disks are single-AZ; always check PV nodeAffinity and node AZ labels before concluding scheduling issues
Confusing mount vs. attach	attached=true in VolumeAttachment means cloud-side attach succeeded; FailedMount events indicate host-side kernel/filesystem mount failure, not cloud attach failure
Overlooking CSI logs for OBS issues	OBS 403 and credential errors are best identified in Everest CSI logs, not in Kubernetes events alone
Premature finalizer removal	Removing kubernetes.io/pvc-protection without verifying no Pod references can cause data loss
Guessing without evidence	When no clear finding matches, output the evidence gap rather than fabricating a conclusion
Skipping environment check	Always run the environment check script before first diagnosis execution