Gateway Watchdog

Monitor OpenClaw Gateway health by detecting abnormal error rates in logs. Use when: (1) setting up Gateway error monitoring, (2) diagnosing repeated API failures (429, quota, timeouts), (3) user asks about Gateway health or error trends, (4) integrating error detection into heartbeat checks. Detects rate-limiting, server errors, auth failures, network errors, and message delivery failures across all channels.

Detect abnormal error patterns in the OpenClaw Gateway before they cause damage. Works with all channels: Telegram, WhatsApp, Discord, Slack, Signal, iMessage, Feishu, and more.

Born from a real incident: a silent try-catch caused 76,744 failed retries in 8 hours — undetected until the API quota was exhausted.

What It Detects

Category	Patterns
Rate limiting	HTTP 429, rate.limit, too many requests
Server errors	HTTP 5xx status codes
Auth/permission	HTTP 401/403, unauthorized, forbidden, token expired
Network errors	ETIMEDOUT, ECONNREFUSED, ECONNRESET, ENOTFOUND, socket hang up
Delivery failures	sendMessage failed, deliver failed, fetch failed
Custom	User-defined via WATCHDOG_EXTRA_PATTERNS env var

Smart Analysis

• Error rate (errors/min) — more meaningful than raw count
• Spike detection — alerts when errors jump 3x+ vs last check
• Error concentration — flags when 80%+ errors are one type (single fault source)

Quick Start

bash scripts/gateway-watchdog.sh check     # silent unless errors exceed threshold
bash scripts/gateway-watchdog.sh verbose   # always outputs full report
bash scripts/gateway-watchdog.sh history   # show monitoring history
bash scripts/gateway-watchdog.sh trend     # last 24h error trend

Heartbeat integration

Add to HEARTBEAT.md:

## Gateway Error Monitoring (every heartbeat)
- Run `~/.openclaw/workspace/skills/gateway-watchdog/scripts/gateway-watchdog.sh check`
- If output is non-empty, report to user immediately
- No output = healthy, skip reporting

Cron (optional)

openclaw cron add \
  --name "gateway-watchdog" \
  --schedule "*/30 * * * *" \
  --task "Run gateway-watchdog.sh verbose. If errors detected, notify user with the report." \
  --channel last

Configuration

All via environment variables:

Variable	Default	Description
WATCHDOG_THRESHOLD	30	Error count that triggers alert
WATCHDOG_WINDOW	30	Monitoring window in minutes
WATCHDOG_SPIKE_RATIO	3	Alert when errors jump Nx vs last check
WATCHDOG_EXTRA_PATTERNS	(empty)	Custom regex patterns (e.g., 99991400|99991403)
WATCHDOG_STATE	~/.local/state/gateway-watchdog/state.json	State file
WATCHDOG_LOG	~/.local/state/gateway-watchdog/history.log	History log

Adding channel-specific patterns

# Feishu-specific error codes
export WATCHDOG_EXTRA_PATTERNS='99991400|99991403|99991404|99991429'

# Telegram-specific
export WATCHDOG_EXTRA_PATTERNS='Too Many Requests|FLOOD_WAIT|bot was blocked'

# Discord-specific
export WATCHDOG_EXTRA_PATTERNS='DiscordAPIError|Missing Permissions|Unknown Channel'

Interpreting Results

🔴 Alert (Chinese locale)

🔴 Gateway 最近 30 分钟出现 150 条异常错误（阈值: 30，5/min）
📈 错误突增: 12 → 150（3倍阈值触发）

错误分类：
  429/限流: 120
  5xx服务端错误: 5
  认证/权限: 0
  网络错误: 5
  消息投递失败: 20

  ⚠️  单一错误类型「429/限流」占比 80%，可能是单一故障源

🔴 Alert (English equivalent)

🔴 Gateway detected 150 errors in the last 30 min (threshold: 30, 5/min)
📈 Error spike: 12 → 150 (3x threshold triggered)

Error breakdown:
  429/Rate-limit: 120
  5xx Server errors: 5
  Auth/Permission: 0
  Network errors: 5
  Delivery failures: 20

  ⚠️  Single error type "429/Rate-limit" accounts for 80%+ — likely a single fault source

💚 Healthy

No output from check mode.

Limitations

• Requires systemd + journalctl (falls back to ~/.openclaw/logs/ on macOS)
• Reactive, not preventive
• Cannot pinpoint which extension is failing — check error details for clues

Security

• Read-only: Only reads logs
• No credentials: No API keys accessed
• No network: No outbound requests
• User state only: State in ~/.local/state/gateway-watchdog/ (XDG standard, no elevated permissions needed)