Server-Mate

Build or extend a lightweight server monitoring and AI operations workflow for Linux hosts running Nginx or Apache, with optional centralized remote monitoring through the BT-Panel (Baota) HTTP API. Use when Codex needs to collect psutil metrics, parse access, error, or auth logs (locally OR pulled

Version: 1.6.1

Use this skill to design or implement a two-plane monitoring system:

• a Python agent that tails local logs OR pulls remote logs through the BT-Panel HTTP API (centralized, agentless), and samples host metrics
• an OpenClaw-side analyzer that aggregates data, explains failures, answers questions, and sends alerts

The same Agent serves both layouts: leave panel_id empty on a site to keep the legacy local-tail path; set panel_id on a site to pull its access / error logs remotely with no probe installed on the target host.

Start

• Confirm the environment first: Linux distribution, Nginx or Apache, PHP-FPM layout, log paths, webhook target, and whether automated actions may touch a live host.
• Keep collection read-only until the user explicitly asks for automation. Add alerting before any auto-ban or auto-heal behavior.
• In OpenClaw deployments, OPENAI_API_KEY is injected by the runtime when AI analysis is enabled. Do not ask the user to export it manually. Treat webhook URLs or tokens in config.yaml as secrets and do not commit them.
• Treat ./data/GeoIP.conf the same way. It may contain MaxMind AccountID and LicenseKey, so keep it local-only and out of Git.
• Prefer MaxMind's official GeoLite2 workflow through ./data/GeoIP.conf and geoipupdate. Treat the built-in public mirror fallback only as an operator-reviewed bootstrap path when no local .mmdb file is present.
• Treat auto-ban and auto-heal as privileged features. They may execute operator-supplied firewall or service restart commands and should stay disabled or dry_run: true until reviewed.
• Use the references progressively instead of loading everything at once:
  • Read references/architecture.md for overall design, component boundaries, and rollout order.
  • Read references/data-contracts.md before defining JSON payloads, storage schemas, metrics, or natural-language query handlers.
  • Read references/ops-playbook.md before implementing thresholds, webhooks, reports, auto-ban, or self-heal logic.
  • Read references/sqlite-schema.md before extending historical storage or report queries.
  • Use scripts/server_agent.py as the collector, daemon entrypoint, and SQLite rollup writer.

Delivery workflow

1. Map the request to one or more tracks.
   • Agent collection
   • Aggregation and storage
   • Alerting and reporting
   • AI diagnosis
   • Guarded remediation
2. Implement the smallest safe slice first.
   • Start with structured access, error, and system events.
   • Add rollup metrics and natural-language answers next.
   • Add webhook alerts after the counters are stable.
   • Enable auto-ban or auto-heal only when thresholds, cooldowns, allowlists, and audit logs already exist.
3. Remote panel connectivity test (mandatory before first collection of any new BT panel). When remote_panels and sites[].panel_id are first wired up, do not jump straight into --once. First prove the auth handshake works end-to-end with a minimal, side-effect-free probe:
   • Instantiate the panel client (build_panel_registry or equivalent).
   • Issue ONE BTPanelClient.exec_shell("echo server-mate-probe") per panel and assert the returned stdout equals server-mate-probe.
   • On BTPanelAuthError: the most common root cause is NTP clock drift between the Agent host and the panel host — surface the hint emitted by the client and have the operator run chronyc tracking / timedatectl status on both ends before re-trying. Also verify api_key_env actually resolves to a non-empty string and that the panel URL includes the correct port.
   • On any other BTPanelError: confirm the panel URL is reachable from the Agent host, the panel's API access switch is enabled, and that no firewall is blocking the port.
   • Only after the probe succeeds for every configured panel may the agent enter normal cron-driven collection.
4. Validate with real or synthetic logs before changing production services.
5. Explain caveats in plain language.
   • Example: UV is often an approximation based on IP and user-agent unless the site provides a stronger visitor key.
   • Example: upload bandwidth is unavailable unless the access log includes request length or a similar field.

Agent rules

• Prefer Python, psutil, and the standard library for the first implementation.
• Prefer a generated ./config.yaml plus local SQLite state such as ./metrics.db before adding external services.
• Keep generated artifacts inside the current skill workspace by default: ./config.yaml, ./metrics.db, ./logs/, and ./reports/. Do not default to /opt, /var/log, or other system-wide directories.
• Prefer the system_metrics + sites[] matrix layout from config.example.yaml instead of new single-site keys.
• Support configurable log paths. Do not hardcode site layouts when the vhost config can be read instead.
• Emit structured JSON with timezone-aware timestamps, host or site identifiers, event type, and enough raw context to debug parser mistakes.
• In multi-site mode, collect host CPU or memory metrics once per cycle and keep site log parsing isolated per domain.
• Separate parsing, aggregation, transport, and action execution so that HTTP push, stdout replay, file drop, or websocket transport can be swapped independently.
• Keep unknown lines and parser failures as first-class counters instead of dropping them silently.
• When the user asks to monitor a remote server that already runs BT-Panel (Baota), do not instruct them to install Server-Mate, a daemon, or any extra package on that target. Instead, guide them to declare the panel under the global remote_panels block in config.yaml and bind the relevant sites[] entry to it via panel_id. The local Agent will pull access / error logs through the BT HTTP API automatically. Always inject the api_key via api_key_env, never as plaintext, and remind the user that config.yaml containing any panel credential must stay out of version control.
• Any code path that builds a remote shell command must route every operator-supplied path through the project's _safe_remote_path helper (or an equivalent: shlex.quote PLUS a structural rejection of NUL / CR / LF / empty input). Never f-string a raw remote_path, glob, or filename into an ExecShell payload, even when the value "obviously" comes from config.yaml — configuration is an attacker-controlled surface in multi-tenant or shared-checkout deployments. The same rule applies to any new BT API helper added later (file move, log rotate, restart command, etc.): validate, then quote, then splice.

Analyzer rules

• Store raw events separately from derived counters.
• Model traffic, performance, security, spider, and error signals as independent reducers over the same event stream.
• Translate natural-language requests into:
  • a time window
  • filters
  • an aggregation
  • a presentation format
• For AI error explanations, pass the fingerprint, surrounding context, and normalized fields instead of dumping entire logs.

Safety rules

• Treat auto-ban and auto-heal as opt-in features.
• Default Guarded Automation to dry_run: true and keep it there until the user has observed automation notifications and audit history for several days.
• Never flip dry_run to false, or enable auto_ban.enabled / auto_heal.enabled, unless the operator explicitly approves the command templates, allowlists, cooldowns, and audit destinations.
• Require cooldowns, max actions per window, and allowlists before running firewall or restart commands.
• Require whitelist checks before any ban command. Never ban loopback, RFC1918 private ranges, or trusted crawler families by default.
• Require TTL-based unban or an equivalent release plan for every ban. Do not create permanent firewall blocks from the first implementation.
• Record an audit event for every alert, dry-run, ban, unban, restart, and failed remediation attempt.
• Store audit history in SQLite tables such as automation_actions and banned_ips, and expose simple lookup queries in user-facing docs.
• Prefer one-shot remediation followed by escalation. Do not loop restarts.

Report expectations

• Daily report: prior-day PV, UV, IP, request totals, bandwidth, status mix, top errors, and slow endpoints.
• Weekly report: blocked IP trends, crawler trends, suspicious route clusters, and recurring slow routes.
• Monthly report: bandwidth peak, disk growth, capacity warning, and remediation summary.

Automation scheduling

Use external scheduling for production unless the user explicitly wants an always-on daemon-only design.

• Recommended ingestion pattern:
  • Run server_agent.py --once every 10 minutes from cron or a systemd timer.
  • This keeps log parsing incremental, writes SQLite rollups, and avoids duplicate resident processes.
• For systemd deployments in Clawhub-style packaging:
  • Do not rely on bundling a .service file inside the skill package.
  • Generate a host-local unit with server_agent.py --config ./config.yaml --generate-service, then paste it into /etc/systemd/system/server-mate.service.
• Recommended report pattern:
  • Run report_generator.py as one-shot scheduled jobs.
  • Daily PDF push at 01:00.
  • Weekly PDF push every Monday at 01:10.
  • Monthly PDF push on day 1 at 01:20.
• In multi-site mode, a single scheduled report_generator.py run should iterate over every configured site unless the user explicitly passes --site.

Release notes for 1.6.1

• Light/Dark/Auto Theme switcher: Exposed a dropdown theme menu at the top right of the dashboard. Seamlessly switch between Light, Dark, and System Auto mode (bound to prefers-color-scheme). Toggling the theme dynamically adapts the Chart.js scales, grid lines, tick fonts, and legends for high-contrast viewing. Persists selection in localStorage.
• PC & Mobile Adaptive Layout: Re-engineered SRE cards using viewport-relative layouts and media queries. Stacks metric rings in a 2x2 grid on tablets and 1x1 on mobile, shrinks margins and text sizes on narrow viewports, and wraps data tables in scrollable wrappers (overflow-x: auto) for smooth swipe operations.

Release notes for 1.6.0

• Built-in SRE Web Dashboard: Integrated a lightweight HTTP server using Python's standard library (http.server). Serves a premium, visually stunning dark-themed real-time console at / featuring SVG circle meters, network rate cards, Top CPU processes, active alert cards, blocked firewall IP tables, and dynamic line charts (CPU & QPS history) using Chart.js.
• AI Security Shield (Dynamic Audit): Integrated LLM-driven traffic audit. Before banning candidate IPs detected by static heuristic thresholds, the Agent compiles request log sequences and queries the LLM for security classification, preventing crawler bans and logging block reasons. Uses caching (llm_shield_cache) in state to save tokens.
• Standalone API Key Configuration: Allowed configuring api_key directly in config.yaml under ai_analysis settings. Enables standalone monitoring cycles and AI diagnosis without external environment dependencies.

Release notes for 1.5.2

• AI Diagnosis Caching (ai_cooldown_seconds): Added persistent caching in state for AI-driven alert diagnoses. Reuses cached diagnoses for the same alert within a 1-hour window (default ai_cooldown_seconds: 3600), reducing token consumption by up to 90%. Prints a 【AI: 缓存】 badge to inform the operator.
• Accurate CPU & Process Sampling: Redesigned collect_system_snapshot() to use a two-pass CPU sampling method (base times calculation -> sleep -> delta calculation) with a 1.0s wait. Prevents python startup and log-parsing transients from causing false CPU warnings and enables correct process CPU percentage measurement.
• CPU-Spike Auto-Remediation (Auto-Ban): Integrated high-CPU warnings (cpu_high / iowait_high) with auto_ban. If CPU is exhausted, the Agent automatically checks for non-whitelisted client IPs driving requests above the rate threshold (cpu_spike_rpm_threshold: 60.0) and issues auto-ban commands (e.g. iptables) to mitigate CC/DDoS attacks on slow endpoints.

Release notes for 1.5.1

• Post-Alert Automatic Diagnostics: Added run_alert_diagnostics() which triggers diagnostic shell commands (e.g., ps, uptime, ss, free, dmesg, df, systemctl status) when hardware/service alerts fire, appending commands output directly to webhooks.
• Alert Recovery Notifications: Added track_alert_recovery() and deliver_recovery_alerts() to track currently firing alerts in state and notify webhooks when resolved (e.g. ✅ Server-Mate 已恢复), including human-readable duration statistics and peak values.
• Diagnostics Configuration: Added configurable settings under diagnostics section (enabled, timeout_seconds, max_lines_per_command, recovery_notification, recovery_min_duration_seconds).

Release notes for 1.5.0

• Deep Linux System Metrics — 4-Layer Expansion: collect_system_snapshot() now collects 28+ metrics across four layers, all via psutil + standard library (zero new dependencies).
  • Layer 1 — CPU detail, memory/swap, disk IOPS, network rate: cpu_user_pct, cpu_system_pct, cpu_iowait_pct; swap_used_pct, swap_used_bytes, memory_used_bytes, memory_available_bytes; per-cycle delta disk_read_iops, disk_write_iops, disk_read_bytes_delta, disk_write_bytes_delta; net_rx_mbps, net_tx_mbps, net_rx_errs, net_tx_errs, net_rx_drop, net_tx_drop.
  • Layer 2 — Process accounting: process_count, process_running, process_sleeping, process_zombie; top_cpu_procs and top_mem_procs (top 5 each).
  • Layer 3 — Inode & extra partitions: disk_inode_used_pct on root; configurable extra_disk_partitions list with per-mount used_pct, free_bytes, and inode_used_pct.
  • Layer 4 — TCP states & systemd service health: tcp_established, tcp_time_wait, tcp_close_wait from psutil.net_connections; configurable service_probes list checked via systemctl is-active (returns service_failed_units).
• Delta I/O tracking via _io_state: a mutable dict passed through run_cycle → collect_system_snapshot persists previous counter values across calls, enabling accurate per-cycle rate metrics without a resident process.
• 10 new alert kinds in evaluate_alerts(): iowait_high, swap_high, memory_critical, net_errors, high_iops, zombie_process, inode_low, disk_multi_low, tcp_timewait_high, service_down — each with a dedicated label, remediation suggestion, and detail block in render_alert_markdown().
• 7 new threshold keys in config.yaml / normalize_config(): iowait_pct (30%), swap_pct (60%), memory_min_available_mb (200 MB), net_error_count (100), disk_write_iops (5 000/s), inode_used_pct (90%), tcp_timewait_count (5 000).
• 4 new system_metrics config options: collect_processes, collect_tcp_states, service_probes, extra_disk_partitions — all default-safe and backwards compatible.
• Zero-downtime DB migration: migrate_schema() is called from init_database() and idempotently ALTER TABLE metric_rollups ADD COLUMN for all 11 new columns using PRAGMA table_info — existing databases are upgraded in-place without data loss.
• 11 new metric_rollups columns: avg_cpu_iowait_pct, avg_swap_used_pct, avg_net_rx_mbps, avg_net_tx_mbps, avg_disk_read_iops, avg_disk_write_iops, max_process_count, max_zombie_count, min_inode_free_pct, avg_tcp_established, max_tcp_timewait — all persisted through upsert_metric_rollup().

Release notes for 1.4.1

• Centralized Remote Monitoring architecture: Server-Mate is now formally a single-Agent-many-hosts collector. Local sites (panel_id empty) and remote sites (panel_id bound to a remote_panels entry) coexist in the same sites[] matrix and share every downstream pipeline (rollups, AI diagnosis, webhooks, PDF reports). Target servers require only that BT-Panel API access is enabled — no Python interpreter, no probe, no daemon, no log shipper.
• Deep BT-Panel API integration:
  • Compliant signing scheme — HMAC-like MD5 signature: request_token = md5(str(request_time) + md5(api_key)), regenerated on every attempt (including retries).
  • Strict POST-only transport per BT official documentation, with auth and business parameters merged into the same application/x-www-form-urlencoded body.
  • requests.Session() per panel for TCP/TLS pooling and BT cookie persistence — eliminates handshake overhead in cron-driven, high-frequency workloads.
• Memory safety — byte-level throughput protection: each cycle issues a single tail -c +<offset> | head -c <chunk> ExecShell call capped at chunk_bytes (default 5 MB). When a remote error_log explodes by hundreds of MB the Agent's memory footprint stays constant, OOM and HTTP timeout are categorically prevented, and residual bytes roll over to subsequent cron ticks. Backlog is stamped into the persisted cursor (backlog_bytes, status="backlog") and emitted as a WARNING.
• Security hardening:
  • Command-injection defense via _safe_remote_path = shlex.quote + structural NUL/CR/LF rejection — applied to every remote path before it is spliced into any ExecShell command. Mandatory for every new BT helper added later.
  • NTP time-drift auto-detection: HTTP 401/403 and HTTP 200 + {"status": false, "msg": "request_token error"} payloads are recognised (English + Chinese variants) and surfaced with the operator-facing hint "Authentication failed. Please check if the time on the Agent server and the Remote BT panel are synchronized (NTP Time Drift)." — both in the raised BTPanelAuthError and in the WARNING log.
• LogReader abstraction (scripts/log_reader.py): LocalLogReader (delegates to the existing inode/truncate-aware function) and BTRemoteLogReader (maintains remote_offset + remote_size, treats remote_size < remote_offset as logrotate). State persists into the existing server_agent_state.json so restarts neither double-read nor lose remote bytes.
• Per-site fault isolation in run_cycle: a single flaky panel can never crash the cron tick or starve other sites; the failure is captured and stamped into the cursor's status field for operator visibility.

Release notes for 1.3.2

• Multi-site matrix config using sites[] plus global system_metrics
• Host-global metrics stored separately from site-local business rollups
• Logrotate-tolerant incremental readers with inode or truncate recovery
• Guarded Automation with dry_run, whitelist checks, TTL-based unban, cooldown-based auto-heal, and SQLite audit trail
• SSH brute-force detection from logs.auth_log with ssh_brute_force alerting and optional linked auto-ban
• SSL certificate expiry inspection in report generation and webhook summaries
• Telegram delivery support for alerts and report notices
• GeoIP official refresh support via local ./data/GeoIP.conf and geoipupdate, with an operator-reviewed public mirror bootstrap fallback
• config.example.yaml and docs updated for MaxMind GeoLite2 setup in the current workspace

Copyable cron examples:

cron

*/10 * * * * /usr/bin/env bash -lc 'python3 ./scripts/server_agent.py --config ./config.yaml --once >> ./logs/server-mate-agent.log 2>&1'
0 1 * * * /usr/bin/env bash -lc 'python3 ./scripts/report_generator.py --config ./config.yaml pdf --range daily --send >> ./logs/server-mate-report.log 2>&1'
10 1 * * 1 /usr/bin/env bash -lc 'python3 ./scripts/report_generator.py --config ./config.yaml pdf --range weekly --send >> ./logs/server-mate-report.log 2>&1'
20 1 1 * * /usr/bin/env bash -lc 'python3 ./scripts/report_generator.py --config ./config.yaml pdf --range monthly --send >> ./logs/server-mate-report.log 2>&1'

Systemd note:

• If the host already standardizes on systemd, prefer Type=oneshot services plus timers for reports.
• Use Restart=always only for the long-running --daemon agent mode.

Example requests

• "Design the ingestion API for Server-Mate."
• "Add 404 burst detection and webhook alerts."
• "Explain today's top 5xx error in plain language."
• "Plan a safe auto-heal flow for repeated 502 responses."