--- name: model-verify-flagos description: | Verify the serving stack with a user-specified target model. Runs twice: first with FlagGems/FlagCX disabled (isolate model-specific errors), then with full multi-chip stack enabled. Diffs the two runs to pinpoint which layer caused any failure. user-invokable: true allowed-tools: "Bash(*) Read Edit Write Glob Grep WebSearch WebFetch AskUserQuestion" --- # Target Model Verification Same layer-peeling approach as env-verify, but with the real target model that may require multi-GPU tensor parallelism. Runs the model twice (without and with multi-chip stack) and diffs results to isolate failures. ## Skill Components ``` model-verify/ ├── SKILL.md # This file — execution flow ├── scripts/ │ └── diff_analysis.py # Compare Run A vs Run B, classify errors (JSON) └── references/ └── multichip-errors.md # Multi-chip error patterns and diff truth table ``` **Reused from env-verify:** - `env-verify/scripts/run_offline_inference.py` — Phase A test (parameterized) - `env-verify/scripts/test_serve_mode.py` — Phase B test (parameterized) - `env-verify/references/error-classification.md` — Layer-based error rules ## Prerequisites - Running container with software stack installed (from `install-stack`) - env-verify completed (at least Phase A passed) - User must provide model path If invoked standalone, ask for container name, vendor, and model path. If invoked from `/flagrelease`, these are passed as context. ## Execution Flow ### Step 1: Get Model Info from User **Ask the user for** (use AskUserQuestion if not provided): 1. **Model path** (required) — local path inside container OR ModelScope/HuggingFace ID 2. **--tensor-parallel-size** (optional) — default to GPU count 3. **Additional vllm args** (optional) Get default TP size: ```bash docker exec python3 -c " import torch; print(torch.cuda.device_count() if torch.cuda.is_available() else 1) " ``` **If user does not provide model path → ask and wait. Do not guess.** ### Step 2: Download Model (if needed) If model path is a remote ID (not starting with `/`): ```bash docker exec python3 -c " from modelscope import snapshot_download snapshot_download('', local_dir='/data/models/') " ``` If local directory, verify `config.json` exists: ```bash docker exec test -f /config.json ``` **Timeout:** 600s for large model downloads. ### Step 3: Run A — WITHOUT Multi-Chip Stack Copy the test scripts from env-verify into the container (if not already there): ```bash docker cp /scripts/run_offline_inference.py :/tmp/ docker cp /scripts/test_serve_mode.py :/tmp/ ``` **Phase A (offline):** ```bash docker exec bash -c ' export USE_FLAGGEMS=0 unset FLAGCX_PATH timeout 300 python3 /tmp/run_offline_inference.py \ --model \ --tp \ --trust-remote-code ' > /tmp/run_a_offline.json ``` **Phase B (serve):** ```bash docker exec bash -c ' export USE_FLAGGEMS=0 unset FLAGCX_PATH timeout 360 python3 /tmp/test_serve_mode.py \ --model \ --tp \ --trust-remote-code \ --health-timeout 300 ' > /tmp/run_a_serve.json ``` ### Step 4: Run B — WITH Full Multi-Chip Stack **Skip logic:** If ALL of FlagGems, FlagTree, FlagCX failed install → skip Run B. Report: "Run B skipped: no multi-chip packages installed." Check install-stack results to decide. **Phase A (offline):** ```bash docker exec bash -c ' export USE_FLAGGEMS=1 export FLAGCX_PATH=/tmp/FlagCX export VLLM_PLUGINS=fl timeout 300 python3 /tmp/run_offline_inference.py \ --model \ --tp \ --trust-remote-code ' > /tmp/run_b_offline.json ``` **Phase B (serve):** ```bash docker exec bash -c ' export USE_FLAGGEMS=1 export FLAGCX_PATH=/tmp/FlagCX export VLLM_PLUGINS=fl timeout 360 python3 /tmp/test_serve_mode.py \ --model \ --tp \ --trust-remote-code \ --health-timeout 300 ' > /tmp/run_b_serve.json ``` ### Step 5: Diff Analysis Copy and run `scripts/diff_analysis.py` to compare the two runs: ```bash docker cp /scripts/diff_analysis.py :/tmp/ docker exec python3 /tmp/diff_analysis.py \ --run-a /tmp/run_a_offline.json \ --run-b /tmp/run_b_offline.json ``` Read `references/multichip-errors.md` to interpret the diff and classify errors. ### Step 6: Produce Report ```json { "status": "PASS | PARTIAL | FAIL", "stage": "model-verify", "model": "", "tensor_parallel_size": 8, "run_a_without_multichip": { "flags": {"USE_FLAGGEMS": "0", "FLAGCX_PATH": "unset"}, "phase_a_offline": "PASS | FAIL", "phase_b_serve": "PASS | FAIL", "output_sample": "...", "errors": [] }, "run_b_with_multichip": { "flags": {"USE_FLAGGEMS": "1", "FLAGCX_PATH": "/tmp/FlagCX"}, "skipped": false, "phase_a_offline": "PASS | FAIL", "phase_b_serve": "PASS | FAIL", "output_sample": "...", "errors": [] }, "diff_analysis": { "conclusion": "BOTH_PASS | MULTICHIP_ERROR | SAME_ERROR | DIFFERENT_ERRORS", "detail": "...", "multichip_component": "FlagGems | FlagTree | FlagCX | plugin | null", "recommended_stack": "full | base | none" } } ``` **`recommended_stack`** — tells downstream skills which stack to use: - `full` — Run B passed (USE_FLAGGEMS=1, FLAGCX_PATH set) - `base` — only Run A passed (USE_FLAGGEMS=0, FLAGCX_PATH unset) - `none` — Run A also failed (model can't serve) **Status logic:** - `PASS` — both Run A and Run B succeed - `PARTIAL` — Run A passes, Run B fails - `FAIL` — Run A fails ## Error Handling | Failure | Behavior | |---------|----------| | Model path not provided | Ask user, wait | | Model path not found | Report exact path, exit with error | | Model too large for memory | Report OOM, suggest reducing TP or dtype | | TP > available GPUs | Report "requested TP=X but only Y available" | | Server hangs | Kill after timeout, capture last logs | | Run A and Run B both fail | Report both errors separately | **Rule:** Run BOTH runs regardless of individual failures. Maximize error coverage. ## Timeout Rules | Operation | Timeout | |-----------|---------| | Model download | 600s | | Phase A (offline) | 300s | | Phase B (serve + test) | 360s |