huawei-cloud-msmodelslim-model-adapt
Create basic Transformers model adapters for msModelSlim. Implements required interfaces and completes a four-step verification workflow: generate test model -> full fallback quantization -> weight verification -> quantization description validation. Use this skill when the user wants to: (1) create msModelSlim adapters for decoder-only LLM, (2) adapt understanding VLM text backbones for quantization, (3) implement W8A8/W4A16 quantization workflow for new models. Trigger: user mentions "msModelSlim", "adapter", "model adapter","quantization", "W8A8","W4A16", "transformers", "LLM", "VLM", "adapter creation", "适配器","模型适配", "量化", "模型适配器", "LLM量化"
Huawei Cloud msModelSlim Model Adapter
Overview
This skill guides how to create basic adapters for new models to run W8A8/W4A16 quantization workflows in msModelSlim.
Architecture: Model Analysis -> Adapter Creation -> Registration -> Verification (4 Steps)
Related Skills:
huawei-cloud-msmodelslim-model-analysis- Model structure analysis before adapter implementationhuawei-cloud-ascend-profiler-db-explorer- Optional: Performance analysis after deployment
Scope
Supported:
- Decoder-only LLM
- Understanding VLM (text/LLM backbone only)
Not supported:
- Multimodal generation (Stable Diffusion/Flux/Wan)
- Encoder-only models
- Non-Transformers architectures
- High-level quantization methods (SmoothQuant/QuaRot/FA3/FlatQuant)
Architecture
┌─────────────────────────────────────────────────────────────┐
│ msModelSlim Model Adapter Skill │
├─────────────────────────────────────────────────────────────┤
│ ┌──────────────────┐ ┌──────────────────────────────┐ │
│ │ Model Analysis │───▶│ Adapter Creation │ │
│ │ - config.json │ │ - LLM Adapter Template │ │
│ │ - modeling_*.py│ │ - VLM Adapter Template │ │
│ └──────────────────┘ │ - Required Interfaces │ │
│ └──────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Registration │ │
│ │ & Installation │ │
│ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Verification (4 Steps) │ │
│ │ 1. Generate Test Model → 2. Full Fallback Quant │ │
│ │ 3. Weight Verification → 4. Quant Description │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Architecture Components
This skill involves the following cloud services and components:
- msModelSlim: Huawei Cloud's model quantization framework for efficient model compression
- Transformers Library: Hugging Face Transformers for model loading and processing
- ModelScope: Model download and management platform
- Ascend NPU: Target hardware for quantized model deployment
Use Cases
Typical Problem Scenarios:
- Need to deploy LLM models with reduced memory footprint on Ascend NPU
- Want to optimize inference speed without significant accuracy loss
- Migrating models that don't have built-in msModelSlim support
- Need W8A8/W4A16 quantization for decoder-only LLM or VLM text backbones
Typical User Phrases:
- "How to quantize my custom LLM model for Ascend?"
- "Create msModelSlim adapter for Qwen model"
- "Implement W4A16 quantization workflow"
- "Adapt my VLM text backbone for quantization"
- "How to add quantization support for new models?"
Core Workflow
1. Preparation
- Download Model: Recommended to use
modelscope downloadfor non-weight files.- Example:
modelscope download --model <org>/<model> --local_dir ./models/<name> --exclude '*.safetensors'
- Example:
- Analyze Model: Read
config.jsonandmodeling_*.pyto confirm structure and implementation.- See: Model Analysis Guide
2. Create Adapter
- Use Templates:
- LLM:
assets/model_adapter_template.py - VLM:
assets/vlm_model_adapter_template.py
- LLM:
- Implement Interfaces: Implement
handle_dataset,init_model,generate_model_visit,generate_model_forward,enable_kv_cache. - Key Principles:
visitandforwardmust be strictly consistent.- MoE models recommended to unpack to pure linear layers.
- See: Implementation Guide
3. Registration & Installation
- Register model and entry in
config/config.ini, then executebash install.sh. - See: Registration Guide
4. Verify Adapter (Required)
- Must execute four-step verification: Generate test model -> Full fallback quantization -> Verify full fallback model matches float weights exactly and can load/save completely -> Verify actual quantization workflow works (including description file rule validation).
- See: Verification Guide
Common Scripts
Scripts located in scripts/ directory:
scripts/step1_generate_test_model.pyscripts/step2_run_quantization.pyscripts/step3_verify_weights.pyscripts/step4_verify_quant_description.py
Prerequisites
System Requirements
- Python 3.8+
- transformers >= 4.40.0
- msmodelslim >= 1.0.0
Environment Check
Prerequisite check: Python3 + transformers + msmodelslim required
bash python3 --version # Python3 >= 3.8 python3 -c "import transformers; print('OK')" # Transformers library python3 -c "import msmodelslim; print('OK')" # msModelSlim libraryIf not installed:
pip3 install --user transformers msmodelslim
Reference Documents
| Document | Description |
|---|---|
| Model Analysis Guide | Model structure analysis guide |
| Implementation Guide | Adapter implementation instructions |
| Registration Guide | Registration and installation guide |
| Verification Guide | Four-step verification workflow |
| Interface Checklist | Required interface implementation checklist |
| Core Workflow | Core workflow documentation |
| Acceptance Criteria | Functional acceptance criteria |
| IAM Policies | IAM permission policies required by this skill |
| Troubleshooting | Common issues and solutions |
Requirements
- transformers >= 4.40.0 installed
- msmodelslim >= 1.0.0 installed
- Transformers model to be adapted
- Understanding of target quantization scheme (W8A8/W4A16)
Core Commands
The four-step verification workflow is executed through the scripts in
the scripts/ directory:
# Step 1: Generate a random-weight test model (slim)
python3 scripts/step1_generate_test_model.py \
--model-path ./models/<model_name> \
--output-path ./test_model
# Step 2: Run full fallback quantization (LLM / use --model-family vlm for VLM)
python3 scripts/step2_run_quantization.py \
--model-path ./test_model \
--output-path ./quantized_model \
--model-type <model_type> \
--model-family llm
# Step 3: Verify fallback weights match the float model exactly
python3 scripts/step3_verify_weights.py \
--original-path ./test_model \
--quantized-path ./quantized_model \
--tolerance 1e-5
# Step 4: Verify the quantization description file against your rules
python3 scripts/step4_verify_quant_description.py \
--desc-path ./quantized_model \
--rules-path ./verify_rules.json
Parameter Confirmation
| Script | Parameter | Required | Description |
|---|---|---|---|
| step1 | --model-path | Yes | Path to the original model directory |
| step1 | --output-path | Yes | Output directory for the generated test model |
| step1 | --num-layers | No | Number of layers for the slim test model (default 2) |
| step2 | --model-path | Yes | Test model directory (step1 output) |
| step2 | --output-path | Yes | Output directory for the quantized model |
| step2 | --model-type | Yes | Model type of the target model |
| step2 | --model-family | No | llm (default) or vlm |
| step2 | --config-path | No | Fallback quantization config; auto-generated if omitted |
| step3 | --original-path | Yes | Directory of the float/step1 model |
| step3 | --quantized-path | Yes | Directory of the step2 quantized model |
| step3 | --tolerance | No | Max allowed weight difference (default 1e-5) |
| step4 | --desc-path | Yes | Quantization output dir or description file path |
| step4 | --rules-path | Yes | Verification rules JSON file path |
- Downloads
- Security audit
- Review
- Last updated
- 1w ago
- Current version
- v1.0.2
- License
- MIT-0
