JarvisLabs GPU

Agent guide for running and monitoring GPU experiments with the jl CLI on JarvisLabs.ai.

JarvisLabs GPU CLI (jl) — Agent Guide

Install The CLI

bash

uv tool install jarvislabs

Alternative:

bash

pip install jarvislabs

Getting Started

Verify auth with jl status --json before doing anything. If not logged in, use jl setup --token <token> --yes. You can also authenticate via export JL_API_KEY="...".

Use --help on any command to discover flags (e.g., jl run --help, jl create --help). If something goes wrong, use jl run logs, jl run status, and jl exec to diagnose — don't guess.

Mental Model

• Machine commands (jl create/list/get/pause/resume/destroy/rename/ssh/exec/upload/download) = GPU instance lifecycle and access.
• jl run = managed job on an instance. Uploads code, sets up a Python environment, runs your script in the background with log tracking.
• jl exec = run any command on an instance. Use for system checks (nvidia-smi, ps, df), debugging failed runs, inspecting files, or any raw shell access. No environment setup, no tracking. This is your escape hatch when jl run doesn't cover your use case.

Instances

Creating

bash

jl create --gpu L4 --storage 100 --yes --json
jl create --gpu L4 --spot --yes --json
jl create --vm --cpu --yes --json

--gpu is required for GPU instances. Use --spot only for GPU containers, not GPU VMs or CPU VMs. CPU VMs are created with --vm --cpu; omit --vcpus/--ram to use the smallest available CPU plan from the backend. Run jl create --help for all available flags.

Lifecycle

Instances have three states that matter: Running (billing active), Paused (compute billing stopped, storage billing continues, data persists), Destroyed (everything deleted).

bash

jl pause <id> --yes --json      # stop compute billing, keep data
jl resume <id> --yes --json     # restart a paused instance
jl destroy <id> --yes --json    # permanently delete

Resume caveats:

• Resume is region-locked — an instance always resumes in its original region. GPU swaps are only possible within that region.
• Resume may return a new machine_id. Always use the returned ID for subsequent operations.
• Spot resume is explicit: pass --spot when you want a paused GPU container to resume as spot. Without --spot, resume is on-demand.
• CPU VM resume uses the CPU VM backend path. Pass both --vcpus and --ram if you want to change CPU size on resume.
• Run jl resume --help for all available flags (GPU swap, storage expansion, rename, etc.).

SSH, exec, upload, and download only work on Running instances.

Regions & GPUs

Valid region codes for new instances: IN1, IN2, EU1.

IN1 is Chennai (india-chennai-01). IN2 is Noida (india-noida-01).

If --region is omitted, the CLI picks a region based on GPU availability.

Use jl resources --json as the source of truth for GPU, region, and workload availability. VM instances require at least one SSH key and a matching workload_type="vm" row.

Container template availability is not exposed in jl resources --json. Current CLI policy allows IN1 container creates only with --template pytorch; other IN1 container templates fail before launch until backend/product support is confirmed.

Run jl gpus to check GPU availability and pricing. Output shows GPU Containers and GPU VMs tables with separate availability for each. Spot prices are shown only for GPU containers.

Run jl resources when you also need CPU VM availability and pricing. It shows GPU containers, GPU VMs, and CPU VMs, with one shared available/unavailable legend at the end.

How to read jl gpus --json availability:

• num_free_devices: GPUs free right now. Spot creates use this count, and require spot_price to be present.
• effective_num_free_devices: GPUs available for on-demand creates on that server, including GPUs currently used by spot instances that can be preempted. If missing, fall back to num_free_devices.
• These counts are per server. They are not the complete regional GPU capacity.
• workload_type tells which launch type the row belongs to:
  • "container" means use it for normal GPU container creates.
  • "vm" means use it for GPU VM creates.
  • null means the same row applies to both containers and VMs.

Ports & Services

Container instances expose default HTTP ports (each gets its own HTTPS URL):

Port	Service
8889	JupyterLab (url field)
7007	IDE (vs_url field)
6006	Available on generic templates like pytorch (endpoints[0])

VM instances (jl create --gpu ... --vm) get SSH-only access. VMs require at least one SSH key registered (jl ssh-key add). Use ssh_command from jl get <id> --json.

To expose a service (FastAPI, Gradio, etc.), bind to 0.0.0.0:6006 — it's accessible via endpoints[0] on generic templates. Use --http-ports "7860,8080" at creation or resume to expose custom ports. Custom port URLs appear in endpoints after the default 6006 entry.

Run jl get <id> --json to find all service URLs (url, vs_url, endpoints).

Managed Runs

How jl run works

jl run uploads your code to an instance, sets up a Python environment, and runs your script in the background with log and exit code tracking. You need either --on <machine_id> (existing instance) or --gpu <type> (creates a fresh instance).

run_id is tracked locally under ~/.jl/runs/. All run management commands (logs, status, stop, list) depend on these local records. Start and monitor runs from the same machine.

Run targets

Target	What happens
train.py	Uploads to <home>/train.py, runs in <home>/ with shared venv at $HOME/.venv
. or ./project with --script train.py	Rsyncs the directory to <home>/<dirname>/, runs inside it with project venv at <home>/<dirname>/.venv
No target, command after --	No upload. Runs from ~. If $HOME/.venv exists (from a previous file run), its bin/ is prepended to PATH so python and pip resolve to venv versions. Otherwise uses system Python.

Only .py and .sh file targets are supported. For other file types, use a directory target or jl upload + jl exec. Directory targets require rsync installed locally.

Note: File targets with the same basename overwrite each other on the remote (e.g., foo/train.py and bar/train.py both land at /home/train.py). Use directory targets for projects with nested structure.

Pass script arguments after --:

bash

jl run train.py --on <id> --json --yes -- --epochs 50 --lr 0.001

Environment & setup

jl run manages a Python venv on the remote instance. Template packages (torch, etc.) are inherited via --system-site-packages — no need to install them. Venvs persist under the remote home directory across pause/resume.

Venv locations:

• File targets: shared instance-level venv at $HOME/.venv. All file runs share it — deps installed for one script are available to all.
• Directory targets: per-project venv at <home>/<dirname>/.venv. Isolated per project.
• Command mode: no venv is created. If $HOME/.venv exists from a previous file run, python and pip automatically resolve to it via PATH prepend.

How dependencies get installed:

• Directory targets — auto-detected. If your directory has requirements.txt or pyproject.toml (with [project]), deps are installed automatically. No flag needed.
• File targets — no auto-detection. Pass --requirements requirements.txt if you need extra packages.
• --requirements <file> — overrides auto-detection. Uploads and installs the specified file instead.
• --setup <command> — runs a shell command before your script (e.g., --setup "pip install flash-attn"). Runs inside the venv for file/dir targets, raw for command mode.

bash

# Directory — auto-detects requirements.txt
jl run . --script train.py --on <id> --json --yes

# Single file — pass requirements explicitly
jl run train.py --on <id> --requirements requirements.txt --json --yes

# Extra setup command
jl run . --script train.py --on <id> --setup "pip install flash-attn" --json --yes

Command mode — when you pass a raw command after -- with no file or directory target. Useful when code already exists on the instance (e.g., uploaded via jl upload, written via jl exec, or left by a previous run). If $HOME/.venv exists from a prior file run, its bin/ is prepended to PATH so python and pip resolve to venv versions. You still get jl run log tracking (logs, status, stop), which is the main advantage over jl exec. --requirements is not supported in command mode.

Important: Command mode runs from ~ (the remote shell home). Use absolute paths or cd explicitly for scripts in specific directories.

bash

jl run --on <id> --json --yes -- python3 /home/train.py
jl run --on <id> --json --yes -- sh -lc 'cd /home && torchrun --nproc_per_node=2 train.py'

Running on an existing instance

bash

jl run train.py --on <machine_id> --json --yes
jl run . --script train.py --on <machine_id> --requirements requirements.txt --json --yes

Lifecycle flags (--keep, --pause, --destroy) are not allowed with --on — the instance is not touched after the run.

Running on a fresh instance

bash

jl run . --script train.py --gpu L4 --keep --json --yes
jl run . --script train.py --gpu L4 --spot --keep --json --yes

Creates a new instance, uploads code, runs the script. Additional flags: --spot (fresh GPU containers only), --vm (VM instead of container, disallows --template and --http-ports), --template (default: pytorch; run jl templates --json to list available), --storage (default: 100GB), --num-gpus (default: 1), --region, --http-ports.

Lifecycle rules for fresh instances:

• With --json or --no-follow: --keep is required. The CLI rejects --pause and --destroy because it returns immediately and cannot apply lifecycle actions later. Use --keep and have the agent pause or destroy the instance after the run completes.
• Without --json or --no-follow (human mode): the CLI stays attached, streams logs, and applies lifecycle when the run finishes. Default lifecycle is --pause.

Use separate jl create when you need to inspect GPU availability, reuse machines across runs, or attach filesystems/scripts.

Monitoring & Control

Reading logs

The primary monitoring command:

bash

jl run logs <run_id> --tail 50

Always use --tail N — without it, the entire log file is returned and can be enormous.

The output includes a header and footer with run state (in non-follow, non-JSON mode):

text

--- run r_abc | machine 123 | running ---

step=100 loss=2.31
step=200 loss=2.11

--- still running | log: /home/jl-runs/r_abc/output.log ---

When done, the footer shows the final state:

text

--- succeeded | exit code: 0 | log: /home/jl-runs/r_abc/output.log ---

Or on failure:

text

--- failed | exit code: 1 | log: /home/jl-runs/r_abc/output.log ---

If the instance is paused, missing, or SSH is unavailable, jl run logs fails before printing any output. Use jl run status <run_id> --json to check those states.

The agent monitoring loop

1. Start detached: jl run ... --json --yes — extract run_id and machine_id from JSON
2. Early check (catch fast failures): sleep 15 && jl run logs <run_id> --tail 30 — if footer says failed, fix and retry immediately
3. Steady-state polling: sleep 120 && jl run logs <run_id> --tail 50
4. Read log body for loss values, errors, or progress
5. Check footer:
   • still running → repeat step 3
   • succeeded | exit code: 0 → download results
   • failed | exit code: N → read error, fix, start a new run

Cadence: 60-120s (short experiments), 180-300s (long training), 300-600s (very long runs).

Checking status

bash

jl run status <run_id> --json

Returns run state, machine_id, exit_code, lifecycle_policy, launch_command, and more. Without --refresh, jl run list shows state as "saved" (a sentinel, not a real run state). Use --refresh or --status to get live state.

Stopping a run

bash

jl run stop <run_id> --json

Kills the entire process group (training script + all child processes). Escalates to SIGKILL if the process doesn't exit after TERM.

System checks via exec

bash

jl exec <id> -- nvidia-smi
jl exec <id> -- ps -ef
jl exec <id> -- df -h

Prefer raw output for jl exec and jl run logs — easier to read and parse. Use --json when you need machine-readable state: create, get, list, run start, run status.

Exit code of the remote command is propagated. For pipes or shell syntax, wrap in sh -lc:

bash

jl exec <id> -- sh -lc 'grep "loss" /path/to/log | tail -5'

File Transfer & Persistence

Upload and download

bash

jl upload <id> ./local /remote          # upload file or directory
jl download <id> /remote ./local         # download file
jl download <id> /remote ./local -r      # download directory

Default destinations: upload without dest → remote home directory. Download without dest → ./<basename> in current local directory.

What persists across pause/resume

The remote home directory (/home/ on containers, /home/<user>/ on VMs) persists. Everything else is ephemeral.

Persists:

• Files and directories under the home directory
• $HOME/.venv (shared venv for file runs) and <project>/.venv (per-project venv for directory runs)
• Attached filesystems (mounted at /home/jl_fs/)
• Run metadata under <home>/jl-runs/<run_id>/

Lost on pause:

• System-level installs (apt-get, global pip packages outside the home directory)
• Files outside the home directory (/tmp, /root, etc.)

Use --setup for system-level reinstalls (e.g., apt-get). Python packages in the venv persist across pause/resume. For recurring system setup, use startup scripts (jl scripts add).

Remote file paths

<home> is /home/ on containers, /home/<user>/ on VMs.

• Uploaded files (via jl run): <home>/<filename> (e.g., train.py → /home/train.py)
• Uploaded directories (via jl run): <home>/<directory_name>/
• Uploaded files (via jl upload): <home>/<filename>
• Shared venv (file runs): <home>/.venv/
• Project venv (directory runs): <home>/<directory_name>/.venv/
• Run metadata: <home>/jl-runs/<run_id>/

Filesystems & supporting commands

Attach a filesystem at creation with --fs-id <id>. Attach a startup script with --script-id <id> (and --script-args). These flags work on both jl create and jl resume.

bash

jl templates --json                    # list available templates
jl ssh-key list --json                 # list registered SSH keys
jl ssh-key add <pubkey-file> --name x  # add SSH key (required for VMs)
jl scripts list --json                 # list startup scripts
jl filesystem list --json              # list filesystems
jl filesystem create --name x --storage 100 --json  # create filesystem

Filesystem caveats:

• Region-bound: A filesystem can only attach to instances in the same region.
• ID changes on edit: Expanding a filesystem (jl filesystem edit) may return a new fs_id. Always use the returned ID.
• The CLI validates that fs_id exists and belongs to the same region before creating/resuming.

Agent Workflow (End-to-End)

bash

# 1. Check GPUs and create instance
jl gpus --json
jl create --gpu L4 --storage 100 --yes --json

# 2. Start detached run
jl run . --script train.py --on <machine_id> --requirements requirements.txt --json --yes

# 3. Early check (catch import/syntax/pip failures fast)
sleep 15 && jl run logs <run_id> --tail 30

# 4. Steady-state monitoring (repeat until footer shows succeeded or failed)
sleep 120 && jl run logs <run_id> --tail 50

# 5. Download results (use /home/<user>/ for VMs instead of /home/)
jl download <machine_id> /home/results ./results -r

# 6. Cleanup
jl pause <machine_id> --yes --json

For fresh instances without a pre-created instance:

bash

# Creates instance inline, runs detached — agent must clean up after
jl run . --script train.py --gpu L4 --keep --json --yes
# ... monitor with jl run logs ...
jl pause <machine_id> --yes --json

Error Handling

When --json is active, CLI validation and API failures are emitted as {"error": "..."} to stdout.

Not all non-zero exits use that shape. jl exec --json returns its own structured payload with stdout, stderr, and exit_code fields.

Agent rule:

• First inspect the JSON shape
• If it has an error key, treat it as a CLI failure
• Otherwise inspect command-specific fields (exit_code, state, run_exit_code)

Anti-Patterns

• Do not use jl run logs --follow — blocks forever, will timeout. --json is also incompatible with --follow.
• Always use --json when starting runs — it returns immediately. Without --json, the CLI streams logs and blocks.
• Do not read full logs without --tail N — can return megabytes of output.
• Do not poll every few seconds — use 60-600s intervals based on expected run duration.
• Do not use lifecycle flags (--keep, --pause, --destroy) with --on — they are rejected. Only for fresh instances.
• Do not use --pause or --destroy with --json for fresh instances — rejected. Use --keep --json and clean up yourself.
• Do not use jl exec for long-running tasks — it blocks until the command finishes. Use jl run which runs in the background with log tracking.
• Do not trust jl run list without --refresh — state shows as "saved" (stale). Use --refresh or --status for live state.
• Do not assume machine_id is stable after jl resume — it may return a new ID. Always use the returned ID.
• Do not forget to pause/destroy instances after experiments — they cost money.

Command Discovery

Every command supports --help for full flag details:

bash

jl create --help       jl run --help       jl ssh-key --help
jl resume --help       jl run logs --help  jl filesystem --help