Local Llm Router

Local LLM model router for Llama, Qwen, DeepSeek, Phi, Mistral, and Gemma across multiple devices. Self-hosted local LLM inference routing on macOS, Linux, and Windows. Local LLM 7-signal scoring engine picks the optimal machine for every local LLM request. OpenAI-compatible local LLM API with context protection, VRAM-aware fallback, and auto-retry. 本地LLM路由 inference router | LLM local enrutador de inferencia. Use when the user wants to optimize local LLM routing, reduce local LLM latency, or load balance local LLM across machines.

Twin Geeks@twinsgeeks

Install

openclaw skills install @twinsgeeks/local-llm-router

Local LLM Router

You are managing a local LLM inference router that distributes local LLM requests across multiple Ollama instances using a 7-signal local LLM scoring engine.

What this local LLM router solves

You have multiple machines with GPUs but your local LLM inference scripts only talk to one. Switching local LLM models between machines means editing configs and restarting. There's no way to compare local LLM latency across nodes, no automatic local LLM failover, and no visibility into which machine handles which local LLM requests.

This local LLM router sits in front of your Ollama instances and picks the optimal device for every local LLM request — based on what local LLM models are hot in memory, how much headroom each machine has, how deep the local LLM queues are, and historical local LLM latency data. Drop-in compatible with the OpenAI SDK and Ollama API.

Setup Local LLM Router

bash

pip install ollama-herd           # install the local LLM router
herd                              # launch the local LLM router (scores and routes)
herd-node                         # launch a local LLM node agent on each device

Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd

Local LLM Router Endpoint

The local LLM router runs at http://localhost:11435 by default. Point any OpenAI-compatible client at http://localhost:11435/v1 for local LLM inference.

python

# local_llm_client — connect to the local LLM router
from openai import OpenAI
local_llm_client = OpenAI(base_url="http://localhost:11435/v1", api_key="not-needed")
local_llm_response = local_llm_client.chat.completions.create(
    model="llama3.3:70b",  # local LLM model
    messages=[{"role": "user", "content": "Hello from local LLM"}],
    stream=True,
)

Local LLM Scoring Engine

Every local LLM request is scored across 7 signals:

Thermal state (+50 pts) — local LLM models already loaded in GPU memory ("hot") score highest
Memory fit (+20 pts) — local LLM nodes with more available headroom score higher
Queue depth (-30 pts) — busy local LLM nodes get penalized
Latency history (-25 pts) — past p75 local LLM latency from SQLite informs expected wait
Role affinity (+15 pts) — large local LLM models prefer big machines
Availability trend (+10 pts) — local LLM nodes with stable availability patterns score higher
Context fit (+15 pts) — local LLM nodes with loaded context windows that fit the estimated token count

Local LLM Context-size Protection

When clients send num_ctx in local LLM requests, the local LLM router intercepts it to prevent Ollama from reloading models unnecessarily:

num_ctx <= loaded context: stripped (local LLM model already supports it)
num_ctx > loaded context: auto-upgrades to a larger loaded local LLM model with sufficient context
Configurable via FLEET_CONTEXT_PROTECTION (strip/warn/passthrough)

Local LLM API Endpoints

Local LLM Fleet Status

bash

# local_llm_fleet_status — all local LLM nodes and queues
curl -s http://localhost:11435/fleet/status | python3 -m json.tool

List all local LLM models across the fleet

bash

# local_llm_model_list — every local LLM model on every node
curl -s http://localhost:11435/api/tags | python3 -m json.tool

Local LLM models currently loaded in memory (hot)

bash

# local_llm_hot_models — local LLM models in GPU memory
curl -s http://localhost:11435/api/ps | python3 -m json.tool

OpenAI-compatible local LLM model list

bash

curl -s http://localhost:11435/v1/models | python3 -m json.tool

Local LLM Request Traces (routing decisions)

bash

# local_llm_traces — recent local LLM routing decisions
curl -s "http://localhost:11435/dashboard/api/traces?limit=20" | python3 -m json.tool

Returns: local LLM model requested, node selected, score breakdown, latency, tokens, retry/fallback status.

Local LLM Model Performance

bash

curl -s http://localhost:11435/dashboard/api/models | python3 -m json.tool

Local LLM Usage Statistics

bash

curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool

Local LLM Fleet Health

bash

curl -s http://localhost:11435/dashboard/api/health | python3 -m json.tool

Local LLM Model Recommendations

bash

curl -s http://localhost:11435/dashboard/api/recommendations | python3 -m json.tool

Local LLM Settings

bash

curl -s http://localhost:11435/dashboard/api/settings | python3 -m json.tool

# Toggle local LLM auto-pull
curl -s -X POST http://localhost:11435/dashboard/api/settings \
  -H "Content-Type: application/json" \
  -d '{"auto_pull": false}'

Local LLM Model Management

bash

# local_llm_model_inventory — per-node local LLM model details
curl -s http://localhost:11435/dashboard/api/model-management | python3 -m json.tool

# Pull a local LLM model onto a specific node
curl -s -X POST http://localhost:11435/dashboard/api/pull \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.3:70b", "node_id": "mac-studio"}'

# Delete a local LLM model from a specific node
curl -s -X POST http://localhost:11435/dashboard/api/delete \
  -H "Content-Type: application/json" \
  -d '{"model": "old-model:7b", "node_id": "mac-studio"}'

Per-app local LLM analytics

bash

curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool

Local LLM Dashboard

Web dashboard at http://localhost:11435/dashboard with eight tabs: Local LLM Fleet Overview, Trends, Local LLM Model Insights, Apps, Benchmarks, Local LLM Health, Recommendations, Settings.

Optimizing Local LLM Latency

Find the slowest local LLM model/node combinations

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT model, node_id, AVG(latency_ms)/1000.0 as avg_secs, COUNT(*) as n FROM request_traces WHERE status='completed' GROUP BY node_id, model HAVING n > 10 ORDER BY avg_secs DESC LIMIT 10"

Check local LLM time-to-first-token

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, model, AVG(time_to_first_token_ms) as avg_ttft FROM request_traces WHERE time_to_first_token_ms IS NOT NULL GROUP BY node_id, model"

Compare hot vs cold local LLM load latency

bash

sqlite3 ~/.fleet-manager/latency.db "SELECT model, CASE WHEN time_to_first_token_ms < 1000 THEN 'hot' ELSE 'cold' END as load_type, AVG(latency_ms)/1000.0 as avg_secs, COUNT(*) as n FROM request_traces WHERE status='completed' AND time_to_first_token_ms IS NOT NULL GROUP BY model, load_type ORDER BY model"

Test local LLM inference

bash

# local LLM via OpenAI format
curl -s http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello from local LLM"}],"stream":false}'

# local LLM via Ollama format
curl -s http://localhost:11435/api/chat \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello from local LLM"}],"stream":false}'

Local LLM Resilience

Auto-retry — re-scores and retries on the next-best local LLM node if failure occurs before the first chunk
Local LLM model fallbacks — specify backup local LLM models; tries alternatives when the primary is unavailable
Local LLM context protection — strips dangerous num_ctx values, auto-upgrades to larger local LLM models
VRAM-aware local LLM fallback — routes to an already-loaded local LLM model in the same category
Zombie reaper — detects and cleans up stuck in-flight local LLM requests
Local LLM auto-pull — pulls missing local LLM models onto the best available node

Local LLM Guardrails

Never restart or stop the local LLM router or node agents without explicit user confirmation.
Never delete or modify files in ~/.fleet-manager/ (contains local LLM latency data, traces, and logs).
Do not pull or delete local LLM models without user confirmation — downloads can be 10-100+ GB.
If a local LLM node shows as offline, report it rather than attempting to SSH into the machine.

Local LLM Failure Handling

Connection refused → local LLM router may not be running, suggest herd or uv run herd
0 local LLM nodes online → suggest starting herd-node on devices
mDNS discovery fails → use --router-url http://router-ip:11435
Local LLM requests hang → check for num_ctx in client requests; verify context protection
Local LLM API errors → check ~/.fleet-manager/logs/herd.jsonl

Local Llm Router

Install

Local LLM Router

What this local LLM router solves

Setup Local LLM Router

Local LLM Router Endpoint

Local LLM Scoring Engine

Local LLM Context-size Protection

Local LLM API Endpoints

Local LLM Fleet Status

List all local LLM models across the fleet

Local LLM models currently loaded in memory (hot)

OpenAI-compatible local LLM model list

Local LLM Request Traces (routing decisions)

Local LLM Model Performance

Local LLM Usage Statistics

Local LLM Fleet Health

Local LLM Model Recommendations

Local LLM Settings

Local LLM Model Management

Per-app local LLM analytics

Local LLM Dashboard

Optimizing Local LLM Latency

Find the slowest local LLM model/node combinations

Check local LLM time-to-first-token

Compare hot vs cold local LLM load latency

Test local LLM inference

Local LLM Resilience

Local LLM Guardrails

Local LLM Failure Handling

Related skills