tensorpool

This skill helps users migrate their local machine learning scripts to run on TensorPool GPU clusters using the interactive cluster workflow (tp ssh). Use th...

Security Scan

The skill's instructions are coherent with migrating ML scripts to TensorPool, but it explicitly directs the agent to autonomously modify and re-run user code and install packages without asking — behaviour that is surprising and potentially risky.

Purpose & Capability

The name/description (migrating local ML scripts to TensorPool via tp ssh) matches the instructions: discovering the tp CLI, creating clusters, transferring code, running on GPU, and debugging. There are no unrelated required env vars, binaries, or install steps declared.

Instruction Scope

SKILL.md tells the agent to 'proactively diagnose and fix the code without asking for permission' and to iterate until the script runs, including running pip installs, editing paths, creating directories, changing code (fixing dataloaders, dtypes, etc.), and re-running. That is scope creep from a user-consent perspective: it grants broad discretion to modify local code and environment automatically. The instructions also expect access to local files, SSH keys, and the user's TensorPool account — reasonable for the task but high-impact actions that should require explicit user approval and safe-guards.

Install Mechanism

This is an instruction-only skill with no install spec and no code files; nothing is written to disk by the skill itself. The runtime guidance to run 'pip install tensorpool' or pip install missing packages is expected for this purpose, but note these commands will alter the user's Python environment if executed.

Credentials

The skill declares no required environment variables or credentials. The instructions assume the user has a TensorPool account and local auth (tp CLI configured) and access to SSH keys, which is proportional. However, because the skill may prompt for or use API keys/SSH credentials at runtime, users should be aware the agent will operate using whatever local credentials or secrets they provide or that are present in the environment.

Persistence & Privilege

always is false and there's no install or self-persistence. The skill can be invoked autonomously (platform default), which increases impact if automated changes are allowed, but autonomy alone is not a disqualifier. The primary concern is the instruction to modify user files without explicit consent rather than persistent privileges.

What to consider before installing

This skill appears to do what it says (migrate scripts to TensorPool), but it instructs the agent to automatically edit, install packages, and re-run your code without asking. Before installing or enabling it: (1) require explicit confirmation before any changes, (2) use a disposable environment (virtualenv/conda or container) and back up your code, (3) be prepared to review all pip installs and code edits, (4) do not expose long-lived API keys/SSH keys unnecessarily — prefer temporary credentials or local authenticated sessions, and (5) monitor cluster creation to avoid unexpected cloud costs. If you need stronger guarantees, ask the skill author to remove the 'fix without asking' directive or to add an explicit opt-in prompt before making changes.

SKILL: Convert Local ML Scripts to TensorPool GPU Production Runs

Overview

This skill helps users migrate their local machine learning scripts to run on TensorPool GPU clusters using the interactive cluster workflow. Use this when a user has a working local script and wants to scale it up to professional GPU hardware (H100, H200, B200, B300, etc.).

Trigger phrases: "run this on H100", "scale to TensorPool", "convert to production", "run on cloud GPU", "migrate to H100", "make this faster on GPU", "run on B200"

Workflow: Analyze local script → Discover CLI commands → Prepare for cluster → Create GPU cluster → Transfer code → Run on GPU → Retrieve outputs → Clean up

---

Error Handling & Iterative Fixing

When running the user's script on TensorPool and it throws errors, Claude should proactively diagnose and fix the code without asking for permission, as long as the fix is in service of the script's original training/inference objective.

Guiding principles:

• Fix it, don't just report it. If a script fails with a traceable error (import errors, CUDA issues, shape mismatches, OOM, path errors, dependency issues), read the traceback, identify the root cause, apply the fix, and re-run.
• Stay in scope. Fixes must serve the original purpose of the script. Fixing a broken dataloader or adjusting batch size for OOM is in scope. Rewriting the model architecture or changing the training objective is not — ask the user first.
• Iterate until it runs. It's normal for scripts to need 2-5 rounds of fixes when moving from local to cloud. Keep going — fix dependency issues, path problems, CUDA compatibility, dtype mismatches, etc. one by one.
• Explain what you changed. After each fix, briefly note what broke and what you did so the user can learn and replicate.

Common auto-fixable errors:

• ModuleNotFoundError → pip install the missing package, re-run
• CUDA out of memory → reduce batch size, enable gradient checkpointing, or switch to LoRA
• FileNotFoundError → fix hardcoded paths, create missing directories (mkdir -p)
• RuntimeError: expected dtype → add .to(device) or fix dtype mismatches (fp32 vs bf16)
• NCCL timeout / distributed errors → fix environment variables, world size, rank config
• KeyError in dataset/config → inspect the data format and adjust loading code
• AttributeError from API changes → update to current library API (e.g., deprecated HuggingFace args)

Out of scope (ask the user):

• Changing the model being trained
• Changing the training objective or loss function
• Significantly altering hyperparameters beyond what's needed for the hardware (e.g., changing learning rate schedule)
• Switching frameworks entirely (e.g., PyTorch → JAX)

---

CLI Discovery (CRITICAL — Always Do This First)

The TensorPool CLI (tp) changes frequently. Never assume command syntax from this document. Always discover current commands at runtime.

Step 0 — Discover Available Commands

Before running any tp commands, always run:

# 1. Check tp is installed
pip show tensorpool || pip install tensorpool

# 2. Discover top-level commands
tp --help

# 3. Drill into relevant subcommands
tp cluster --help
tp ssh --help
tp storage --help
tp me --help

Use the output of --help to determine:

• Exact command names and syntax
• Required vs optional arguments
• Available flags and their current names
• Available instance types

If a command fails or the syntax has changed, re-run --help on that subcommand to get the current usage.

General expectations (these may change — always verify with --help):

• There will be commands to: create clusters, list clusters, get cluster info, destroy clusters, SSH into instances, manage storage, and check account info
• Cluster creation will need at minimum an instance type (e.g., 1xH100, 8xB200)
• Multi-node clusters may only be supported for certain instance types
• SSH will take some form of instance or cluster identifier
• Cluster destruction will take a cluster identifier

---

Prerequisites (Verify Before Starting)

1. TensorPool CLI installed:

   pip show tensorpool || pip install tensorpool
   

2. Authenticated:

   # Run the account info command (discover exact syntax via tp --help)
   # If not authenticated, configure your API key from:
   # https://tensorpool.dev/dashboard
   

3. SSH key available (may be needed for rsync/scp to cluster):

   ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519
   

4. User has a TensorPool account — sign up at tensorpool.dev

---

Step-by-Step Migration Workflow

Step 1 — Analyze the Local Script

Before migrating, understand:

1. What does the script do? (training, inference, data processing)
2. What are the dependencies? (check imports, requirements.txt)
3. What data does it need? (local files, datasets, pretrained models)
4. What outputs does it produce? (model checkpoints, logs, results)
5. GPU requirements: Single GPU? Multi-GPU? Memory needs?

Key questions to ask:

• Is there a requirements.txt? If not, create one.
• Are there hardcoded local paths that need adjustment?
• Does it use relative imports that might break?
• Are there large data files that need to be transferred?

Step 2 — Prepare the Script for Cloud Execution

2.1 Create/verify requirements.txt:

# If it doesn't exist, create it
pip freeze > requirements.txt

# Or manually list dependencies
echo "torch>=2.0.0" > requirements.txt
echo "transformers>=4.30.0" >> requirements.txt

2.2 Check for environment variables:

# Create .env file if needed
cat > .env << EOF
HUGGINGFACE_TOKEN=hf_your_token_here
WANDB_API_KEY=your_wandb_key_here
EOF

2.3 Test locally first (if possible):

# Quick sanity check with minimal data
python your_script.py --max_samples 10 --num_epochs 1

2.4 Identify files to transfer:

• Scripts (.py files)
• Configuration files (.yaml, .json, .toml)
• Requirements (requirements.txt)
• Small data files (< 1GB)
• Environment variables (.env)

2.5 Identify files NOT to transfer:

• Large datasets (download directly on cluster)
• Previous outputs/checkpoints
• Git history (.git/)
• Python cache (__pycache__/, *.pyc)
• Virtual environments (venv/, env/)

Step 3 — Create GPU Cluster

Choosing an instance type:

GPU	Memory	Best For
H100	80GB	General training, fine-tuning 7B-70B models
H200	141GB	Large context, 70B+ models, memory-bound workloads
B200	192GB	Latest gen, native FP4/FP6, fastest training
B300	Latest	Newest generation, highest performance
L40S	48GB	Inference, smaller training workloads

Instance types come in configurations like 1xH100, 2xH100, 4xH100, 8xH100, etc.

To create a cluster:

1. Run tp cluster --help to see the exact create syntax and required arguments
2. Choose your instance type
3. Create the cluster and note the cluster/instance identifiers from the output

Wait for provisioning: Clusters typically take 1-2 minutes to become ready.

Step 4 — Get Cluster Information

Use the cluster list/info commands (discovered via tp cluster --help) to get:

• Cluster ID
• Instance ID(s) — needed for SSH
• IP address — needed for rsync/scp
• Status (should be RUNNING)

Step 5 — Transfer Code to Cluster

Recommended: Use rsync (fast, resumable, efficient)

rsync -avz \
  --exclude=".git" \
  --exclude="__pycache__" \
  --exclude="*.pyc" \
  --exclude="venv/" \
  --exclude="outputs/" \
  ./ ubuntu@<cluster-ip>:~/my-project/

Alternative: Use scp for single files

scp script.py ubuntu@<cluster-ip>:~/
scp -r ./data/ ubuntu@<cluster-ip>:~/data/

Step 6 — SSH into Cluster

Use the SSH command discovered via tp ssh --help, passing the appropriate instance or cluster identifier.

You're now on the GPU cluster! All subsequent commands run on the cluster.

Step 7 — Setup Environment on Cluster

7.1 Verify GPU:

nvidia-smi

Expected output: GPU(s) listed with available memory

7.2 Navigate to your project:

cd ~/my-project
ls -la  # Verify files transferred

7.3 Install dependencies:

# Using requirements.txt
pip install -r requirements.txt

# Or install packages individually
pip install torch transformers datasets accelerate

7.4 Set environment variables (if needed):

# Load from .env
export $(cat .env | xargs)

# Or set manually
export HUGGINGFACE_TOKEN=hf_your_token_here
export CUDA_VISIBLE_DEVICES=0

7.5 Download large datasets (if needed):

# Download directly on cluster (faster than transferring)
wget https://example.com/large-dataset.tar.gz
tar -xzf large-dataset.tar.gz

# Or use Hugging Face datasets (downloads automatically)
# No action needed - will download when script runs

Step 8 — Run Your Script on GPU

8.1 Quick test first:

# Run with minimal data to verify everything works
python your_script.py --max_samples 100 --num_epochs 1

8.2 Full production run: Often times the user will input information to the prompt of how to do production run

# Option 1: Direct execution
python train.py --num_epochs 5 --batch_size 32

# Option 2: Use screen/tmux (survives SSH disconnection)
screen -S training
python train.py --num_epochs 5
# Press Ctrl+A then D to detach
# Reconnect later with: screen -r training

# Option 3: Use nohup (runs in background)
nohup python train.py > training.log 2>&1 &
tail -f training.log  # Monitor progress

8.3 Monitor progress:

# Watch GPU usage
watch -n 1 nvidia-smi

# Check logs
tail -f training.log

# Monitor disk space
df -h

Step 9 — Retrieve Outputs

From your LOCAL machine (open new terminal):

CLUSTER_IP=<ip-from-cluster-info>

# Download outputs
rsync -avz ubuntu@$CLUSTER_IP:~/my-project/outputs/ ./outputs/

# Download specific files
rsync -avz ubuntu@$CLUSTER_IP:~/my-project/model.pt ./
rsync -avz ubuntu@$CLUSTER_IP:~/my-project/logs/ ./logs/

Pro tip: Use --progress flag to see transfer progress:

rsync -avz --progress ubuntu@$CLUSTER_IP:~/my-project/outputs/ ./outputs/

Step 10 — Destroy the Cluster

** CRITICAL: Always delete the cluster to avoid charges**

Use the cluster destroy command (discovered via tp cluster --help), passing the cluster identifier.

Verify deletion by listing clusters again — it should no longer appear.

Cost reminder: Clusters bill continuously until destroyed!

---

NFS Storage (Persistent Volumes)

TensorPool offers NFS storage for persistent data across cluster lifecycles and for sharing data across multi-node clusters. Discover the exact commands via:

tp storage --help

---

Resources

• TensorPool Docs: https://docs.tensorpool.dev
• Cluster Quickstart: https://docs.tensorpool.dev/clusters-quickstart
• Instance Types: https://docs.tensorpool.dev/resources/instance-types
• CLI Reference: https://docs.tensorpool.dev/cli/overview
• Community Slack: https://tensorpool.dev/slack
• Contact: [e-mail]

---