Canary Deploy

Safe system changes with automatic baseline capture, canary testing, and rollback for critical infrastructure modifications. Use when making changes to SSH config, firewall rules, network settings, systemd services, kernel parameters, or any system change that could break remote access. Prevents lockouts by validating connectivity before and after changes. Born from a real incident where AllowTcpForwarding=no killed VPN tunnel access.

Safe system changes with pre-flight checks, validation, and automatic rollback.

The Problem

System changes can lock you out:

• SSH hardening breaks remote access
• Firewall rules block needed ports
• Kernel parameters cause instability
• Service restarts break dependencies

Recovery without physical access is painful or impossible.

Quick Start

Before any critical change

bash

# Capture baseline (connectivity, services, ports)
bash scripts/canary-test.sh baseline

# Make your change
sudo nano /etc/ssh/sshd_config

# Validate change didn't break anything
bash scripts/canary-test.sh validate

# If validation fails:
bash scripts/canary-test.sh rollback

For automated changes

bash

# Full pipeline: baseline → apply → validate → rollback-if-failed
bash scripts/critical-update.sh \
  --name "SSH hardening" \
  --backup "/etc/ssh/sshd_config" \
  --command "sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config && sudo systemctl reload sshd" \
  --validate "ssh -o ConnectTimeout=5 localhost echo ok"

Protocol A+B (Manual Workflow)

For interactive sessions where you want human-in-the-loop:

Protocol A: Test interactively

1. Tell the human: "Open a second SSH session as backup"
2. Apply change in the first session
3. Ask: "Test connectivity from the second session"
4. If it works → confirm
5. If it fails → rollback from the backup session

Protocol B: Backup first

1. Run bash scripts/canary-test.sh baseline
2. Verify backup is valid
3. Apply change
4. Run bash scripts/canary-test.sh validate
5. If validation fails → bash scripts/canary-test.sh rollback

Always use both A + B together for maximum safety.

What Gets Checked

Baseline capture

• SSH connectivity (local + remote)
• Open ports (ss -tlnp)
• Running services (systemctl)
• Firewall rules (ufw/iptables)
• Network routes
• DNS resolution
• Config file checksums

Validation

• All baseline checks re-run
• Diff against baseline
• Any regression = FAIL

Critical Change Categories

Category	Risk	Example	Recovery
SSH config	🔴 HIGH	sshd_config changes	Backup session
Firewall	🔴 HIGH	UFW/iptables rules	Pre-change snapshot
Network	🔴 HIGH	Interface/routing changes	Console access
Services	🟡 MEDIUM	systemd unit changes	systemctl restart
Kernel params	🟡 MEDIUM	sysctl changes	Reboot to defaults
Packages	🟢 LOW	apt install/upgrade	apt rollback

References

See references/incident-report.md for the real incident that inspired this skill.