Use this skill for MaxFrame SDK development and documentation navigation on Alibaba Cloud MaxCompute (ODPS). Helps answer MaxFrame API, concept, official example, and supported pandas API questions; create data processing programs; read/write MaxCompute tables; debug jobs (remote or local); and build custom DPE runtime images. Trigger when users mention MaxFrame, MaxCompute with MaxFrame, ODPS table processing, DPE runtime, MaxFrame docs/examples, DataFrame/Tensor operations, or GPU runtime setup. Works for both English and Chinese queries about Alibaba Cloud data processing with MaxFrame.
Use APIs from: maxframe.dataframe, maxframe.tensor, maxframe.learn, maxframe.session, maxframe.udf, maxframe.config. Use canonical imports: import maxframe.dataframe as md and from maxframe.session import new_session; never use from maxframe import new_session.
2. DO NOT Read Private .env Files
Use dotenv.load_dotenv() programmatically. Never read .env files directly with Read tool.
3. Lazy Execution
MaxFrame is lazy: operations build a graph and run only when .execute() is called. Execute only the final result/write action; do not call .execute() on intermediate DataFrame or Series variables unless the user explicitly asks for preview/debug output.
4. Session Management
Always create session before operations, destroy in finally block for cleanup.
5. Operator Selection with User Confirmation
Before implementing processing logic, confirm operator selection with user using scripts/lookup_operator.py. If the user already named the exact operator or asked for code only, write Operator confirmed via user prompt: <operator> before implementation.
6. Documentation Source Order
For MaxFrame documentation questions, check the official online docs first because APIs may change. Use bundled local docs as an offline fallback, for quick cross-checks, or when the website lacks detail.
Do not finish Scenario 0 unless the last non-empty line starts with Sources: and contains a full https://maxframe.readthedocs.io/ URL; never output .... If official docs cannot be checked, include Official docs unavailable: <reason>; using local fallback. before that line.
Red Flags
Thought
Reality
"This is just a simple MaxFrame question"
Questions are tasks. Invoke the skill.
"I already know the MaxFrame API"
Skills have latest patterns. Use them.
"Let me just write the code directly"
Operator selection is MANDATORY.
"I can skip operator confirmation"
User confirmation is REQUIRED.
Scenario 0: Documentation Navigation
Use for API/concept/example questions that do not require a new MaxFrame program.
Workflow Steps
Classify Question — API/operator, concept, supported pandas API, troubleshooting, runtime image, or example
Answer Grounded in Sources — final non-empty line must be Sources: <full official URL> (Primary); append | <local path> (Fallback/Cross-check) only if local docs were used
Table Not Found — check table name and permissions
Timeout Errors — check logview, optimize query
Type Mismatch — check DataFrame dtypes
SQL Errors — review generated SQL in logview
See:references/remote-debug-guide.md for detailed solutions.
Scenario 3: Local Debug Mode
Workflow Steps
Understand Requirements — UDF logic, sample data schema, IDE preference
Create Local Debug Setup — session with debug=True, sample data with md.DataFrame(pd.DataFrame(...))
Provide IDE Setup Guidance — breakpoint setup, execution flow, and final execute only
Example Code Structure
python
import maxframe.dataframe as md
from maxframe.session import new_session
import pandas as pd
session = new_session(debug=True)
sample_data = pd.DataFrame({
'user_id': ['u1', 'u2', 'u3'],
'level': ['gold', 'silver', 'bronze'],
'amount': [1000, 500, 100]
})
df = md.DataFrame(sample_data)
def calculate_discount(row):
# Set breakpoint here in IDE
if row['level'] == 'gold':
return row['amount'] * 0.1
return row['amount'] * 0.02
try:
result = df.apply(calculate_discount, axis=1)
result.execute()
finally:
session.destroy()
See:references/local-debug-guide.md for complete guide.
Scenario 4: Create Custom Runtime Image
Build custom Docker images through conversational guidance using best practices from reference guides.
When to Create Custom Runtime
Create when: need Python libraries not in standard DPE runtime, GPU-enabled processing, specific Python version, custom system dependencies
NOT needed when: standard packages suffice, no GPU requirements
Conversational Workflow
If the user already specifies base image, Python versions, GPU requirement, packages, or output directory, restate those choices and continue. Ask only for missing, ambiguous, or incompatible choices.
Read Best Practices Guide — references/runtime-image-guides/README.md
Base Image Selection — Ubuntu 22.04 (GPU/ML workloads) or Ubuntu 24.04 (modern development)
Python Version Selection — Python 3.11 (production), 3.10-3.12 (development), or all versions
GPU Configuration — CUDA 12.4 + PyTorch 2.6.0+cu124 (if ML workloads)
Iterative Package Collection — collect required packages, note version constraints
Create Support Files — README.md, .dockerignore, requirements.txt
Provide Build and Test Instructions
MaxFrame Usage Example
Step-by-Step Guidance
Step 1: Base Image Selection (ask if missing)
Present Ubuntu options with trade-offs:
text
Which Ubuntu version for your custom runtime?
A. Ubuntu 22.04 (Recommended for most cases)
- Stable, production-ready
- Excellent CUDA support (12.4, 12.1, 11.8)
- Widely tested ML libraries (PyTorch, TensorFlow)
- LTS until 2027
B. Ubuntu 24.04 (Modern/latest)
- Newer system packages
- Latest LTS (until 2029)
- Better for non-GPU workloads
- Python 3.12 integration
Recommendation:
- GPU/ML workloads → Ubuntu 22.04
- Modern development → Ubuntu 24.04
Step 2: Python Version Selection (ask if missing)
text
Which Python versions?
A. Python 3.11 only (Recommended for production)
- Best performance
- Smallest image (~1 GB)
- Excellent package support
B. Python 3.10, 3.11, 3.12 (Development)
- Good compatibility
- Medium size (~2 GB)
- Recent versions
C. All versions 3.7-3.12 (Maximum flexibility)
- Largest image (~3-5 GB)
- Maximum compatibility
- Testing across versions
Recommendation:
- Production → Single version (3.11)
- Development → Recent versions (3.10-3.12)
Step 3: GPU Configuration (ask if missing)
If user mentions GPU or ML packages:
text
Need GPU support?
A. Yes - GPU-enabled with CUDA 12.4 (Recommended)
- Install PyTorch 2.6.0+cu124
- CUDA toolkit 12.4
- Note: Requires Ubuntu 22.04 for best compatibility
B. No - CPU only
- Standard package installation
- Smaller image size
Recommendation: For ML/AI workloads, GPU support significantly improves performance.
Compatibility Handling:
If user selected Ubuntu 24.04 earlier and now requests GPU support:
Explain: "Ubuntu 24.04 has limited CUDA support. Ubuntu 22.04 is recommended for GPU workloads."
AskUserQuestion: "Should I use Ubuntu 22.04 instead for better GPU compatibility?" (Yes recommended)
Step 4: Build Dockerfile Section-by-Section
For each section:
Read pattern from best practices guide
Explain purpose and trade-offs
Write section with inline comments
Accumulate into complete Dockerfile
Sections:
Header — Image metadata, configuration summary
Base setup — FROM, apt packages, locales, timezone
Verification — Health checks, Python version verification
Step 5: Provide Build and Test Instructions
bash
# Build
docker build -t <image-tag> <output-dir>
# Test Python
docker run --rm <image-tag> conda run -n py311 python --version
# Test GPU (if applicable)
docker run --rm --gpus all <image-tag> python -c "import torch; print(torch.cuda.is_available())"
# Test packages
docker run --rm <image-tag> conda run -n py311 python -c "import transformers; print(transformers.__version__)"
# Push to registry
docker push <image-tag>
Step 6: MaxFrame Usage Example — must include new_session(odps=odps_connection, image="..."); never show new_session(image=...) alone.
python
from maxframe.session import new_session
session = new_session(odps=odps_connection, image="your-registry/your-image:v1")
# Your MaxFrame operations here
Default Recommendations
Component
Recommendation
Base Image
Ubuntu 22.04 (production, GPU, ML)
Python
3.11 (production), 3.10-3.12 (development)
GPU
Ubuntu 22.04 + CUDA 12.4 + PyTorch 2.6.0+cu124
Critical Notes
MaxFrame SDK NOT in Runtime Image: SDK and pyodps are client-side only. Custom runtime needs user-specific packages (transformers, pandas, etc.).
See:references/runtime-image-guides/ for detailed guides on base image selection, Python environment strategy, package management, GPU/CUDA configuration, Dockerfile templates, and testing/validation.
Operator Selection Workflow
MANDATORY before implementing processing logic when user mentions specific operations, asks about efficiency/performance, or you need to find appropriate MaxFrame operator.
For documentation-only answers, user confirmation is not required; still use the lookup script to ground API claims.
If the user explicitly names the operator or asks to skip interaction, output Operator confirmed via user prompt: <operator> and implement directly.
Workflow
Identify Operations — list required transformations
