linux-firewall-hardening

Safe Linux firewall hardening with backend detection, idempotent atomic rules, rollback protection, and AI-executable state-machine workflows. Covers ufw, firewalld, nftables, iptables, Docker, Kubernetes CNI awareness, and fail2ban with compliance mapping to CIS/PCI-DSS/SOC2.

When to Use

• Check if a Linux server has active firewall protection.
• Enable and configure a firewall without locking yourself out of SSH.
• Audit existing rules, troubleshoot connectivity, or apply a security profile.
• Automate firewall hardening via an AI agent or CI/CD pipeline.

When NOT to Use

Condition	Alternative
Kubernetes worker node	Use NetworkPolicies / CiliumNetworkPolicy
Firewall managed by Terraform/Ansible/Puppet/Chef	Update IaC source of truth
Cloud workload with Security Group / NSG only	Use cloud provider's firewall API
Inside a container	Escalate to host operator
WSL2, macOS, or shared/managed hosting	See references/special-environments.md

Support files: scripts/audit-firewall.sh (run first), scripts/firewall-plan.sh (dry-run), scripts/firewall-verify.sh (post-apply), scripts/container-port-audit.sh (Docker DNAT detection), scripts/ip-consistency.sh (IPv4/IPv6 drift check). firewall-apply.sh is fully documented in references/firewall-apply.md. Detailed backend guides, Docker/K8s policies, observability, compliance, and recovery are in references/.

🚨 Emergency: I'm Locked Out — What Now?

If you just applied firewall rules and lost SSH connectivity:

1. Wait 5 minutes — the auto-rollback timer (scheduled during VALIDATE) will restore access. Don't panic and don't take destructive actions.
2. Use your second SSH session — if you opened one (pre-flight checklist), switch to it and fix the rules manually.
3. Cloud serial console — AWS EC2 Serial Console, GCP Serial Port, Azure Serial Console, or hypervisor VNC/IPMI/iDRAC.
4. Restore from backup via console — once connected: sudo iptables-restore < ~/firewall-backup-*/iptables-v4.rules
5. Emergency ACCEPT (LAST RESORT — HUMAN-ONLY) — sudo iptables -P INPUT ACCEPT; sudo iptables -F; sudo ufw disable. This exposes the host completely. NEVER auto-execute this command. The agent must refuse to run this autonomously. Only a human operator may issue this via serial console. Re-harden immediately afterward.

Full procedures: references/recovery.md.

---

Prerequisites

• Root or sudo access.
• An active SSH session (risk of lockout).
• Know which ports your services use.

---

NEVER DO (14 Rules)

1. Never flush iptables/nftables on Kubernetes nodes. CNI plugins manage netfilter.
2. Never run iptables -F or nft flush ruleset without a verified backup. Docker/K8s networking will break.
3. Never disable firewalld and use raw iptables simultaneously. Undefined behavior.
4. Never set DROP policy on INPUT before allowing your current SSH port. Immediate lockout.
5. Never disable Docker's iptables management without replacement NAT/routing rules.
6. Never restart networking.service or NetworkManager remotely without console access.
7. Never apply cloud SG and host firewall changes simultaneously without testing.
8. Never enable logging on high-traffic DROP rules without limit rate. Disk flood.
9. Never manage nftables/iptables directly when ufw or firewalld owns the policy. Split-brain state.
10. Never apply outbound default-deny without explicitly allowing DNS, NTP, package mirrors.
11. Never restore firewall backups from a different host, kernel version, or backend mode.
12. Never assume IPv4 rules protect IPv6. Verify both stacks separately.
13. Never change sysctl hardening values on K8s/CNI hosts without explicit CNI profile support.
14. Never enable verbose packet logging without rate limits and log rotation.

---

State Machine

Follow states in order. Do not skip.

text

DETECT → SELECT → PLAN → CONFIRM → VALIDATE → APPLY → VERIFY → (COMMIT | ROLLBACK)

CONFIRM is mandatory. The agent may never self-transition through CONFIRM. See the CONFIRM state section below for the exact gating rules.

State: DETECT

Run the audit script:

bash

bash scripts/audit-firewall.sh           # Human-readable
bash scripts/audit-firewall.sh --json    # Machine-readable

Key outputs: confidence, risk_tier, recommended_backend, halt_reasons, k8s_node, iac_owner.

Risk Tiers & Confidence Gating

Tier	Confidence	Agent Behavior
auto	≥ 90%	Proceed automatically to PLAN
confirmed	70–89%	Proceed but require human confirmation before APPLY
manual	50–69%	Audit-only mode. Generate recommendations, do not apply.
halt	< 50%	Stop immediately. Escalate findings to operator.

Additional halt triggers (regardless of confidence): containerized, K8s node, IaC managed, no rollback mechanism available.

Decision Tree

Condition	Path	Detail
Risk tier = halt	STOP	Resolve blockers first
Inside container	STOP	Escalate to host operator
K8s node detected	STOP	references/k8s-policy.md
Ubuntu/Debian + ufw active	Phase: UFW	references/backend-ufw.md
ufw + firewalld both active	STOP	Resolve conflict
RHEL/Rocky/Alma + firewalld active	Phase: firewalld	references/backend-firewalld.md
nftables active, no frontend	Phase: nftables	references/backend-nftables.md
iptables only	Phase: iptables	references/backend-iptables.md
Docker host	Apply Docker Hardening after phase above	references/docker-hardening.md

Ownership Boundary

Before modifying rules, verify no IaC tool manages the firewall. If Terraform/Ansible/Puppet/Chef/cloud-init is detected → do not mutate. Update the source of truth instead. Full detection logic is in scripts/audit-firewall.sh.

---

State: SELECT

Optionally load a pre-built security profile (references/security-profiles.md):

Profile	Use Case
public-web-server	Open 22, 80, 443. Rate-limit SSH.
internal-database	SSH from mgmt subnet only. DB port from app subnet only.
bastion-host	SSH only. Aggressive rate limiting.
zero-trust-node	Default deny all inbound and outbound.

Or use declarative YAML (references/declarative-policy.md):

text

Imperative (state machine) → Ad-hoc hardening, incident response
Declarative (YAML)        → GitOps, multi-host, reproducible
Mixed                     → YAML as source-of-truth, state machine for verification

---

State: PLAN

Generate a dry-run diff before applying:

bash

bash scripts/firewall-plan.sh --profile public-web-server
bash scripts/firewall-plan.sh --ports 22,80,443
bash scripts/firewall-plan.sh --json     # Machine-readable diff with approval_token
bash scripts/firewall-plan.sh --refresh-audit --json  # Force re-audit + plan

Review the output. If risk_tier is confirmed, present the plan and wait for human confirmation before APPLY.

Plan JSON schema (matches firewall-plan.sh --json output):

json

{
  "backend": "ufw",
  "active_frontend": "ufw",
  "profile": "public-web-server",
  "target_ports": [22, 80, 443],
  "diff": {
    "add":    [{"port": 80, "proto": "tcp", "source": "any"}],
    "skip":   [{"port": 22, "proto": "tcp", "reason": "already_exists"}],
    "remove": []
  },
  "risk_assessment": "low",
  "estimated_disruption": "none",
  "approval_token": "sha256:abc123...",
  "audit_cached": false,
  "audit_cache_file": "/tmp/firewall-audit.json"
}

Approval gate: PLAN output includes an approval_token (hash of plan content). APPLY must be called with --approved-plan=<token>. Token mismatch → exit code 41. This forces explicit human confirmation before Apply.

Audit caching: firewall-plan.sh internally calls audit-firewall.sh --json and caches to /tmp/firewall-audit.json (TTL 5 min). Use --refresh-audit to force refresh.

---

State: CONFIRM (Mandatory — No Exceptions)

This gate prevents the agent from applying firewall rules without explicit human approval.

Why this exists: The agent could (a) invoke firewall-apply.sh without the human seeing the plan, or (b) the human "approves" a stale plan that changed between approval and execution. CONFIRM defeats both — including the agent hallucinating an approval.

Rules (HARD — do not violate):

1. After PLAN, present the full ruleset diff AND the approval token printed by firewall-plan.sh --json (field: approval_token).
2. STOP. Do not proceed in the same response. Do not type the token on the user's behalf.
3. You may ONLY invoke APPLY if the user's most recent message contains that exact approval token. Never infer approval from phrases like "looks good" or "go ahead".
4. APPLY always runs with auto-rollback armed. Committing (disarming rollback) requires a second explicit user instruction after VERIFY passes.
5. A plan token mismatch at APPLY time produces exit code 41 — this is by design. Never bypass it.

Enforcement mechanism: The token is a hash of the plan contents. If the plan changes (e.g., network state drifted between PLAN and APPLY), the token no longer matches → apply is refused. This converts a fuzzy judgment ("did the user approve?") into a cryptographic check the agent cannot rationalize around.

Concrete flow:

text

Agent: PLAN output → shows rule diff + PLAN-TOKEN: a1b2c3d4e5f6
Agent: STOPS. Waits.
User:  "Approved. a1b2c3d4e5f6"
Agent: Runs APPLY with --approved-plan=a1b2c3d4e5f6
Agent: After APPLY → runs VERIFY
Agent: STOPS. Shows VERIFY results.
User:  "VERIFY passed. Commit."
Agent: Runs COMMIT (disarms rollback timer)

---

State: VALIDATE

1. Create Backup (Mandatory)

bash

BACKUP_DIR="$HOME/firewall-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

sudo iptables-save > "$BACKUP_DIR/iptables-v4.rules" 2>/dev/null || true
sudo ip6tables-save > "$BACKUP_DIR/iptables-v6.rules" 2>/dev/null || true
sudo nft list ruleset > "$BACKUP_DIR/nftables.rules" 2>/dev/null || true
sudo ufw status verbose > "$BACKUP_DIR/ufw-status.txt" 2>/dev/null || true
sudo firewall-cmd --list-all --zone=$(sudo firewall-cmd --get-default-zone) > "$BACKUP_DIR/firewalld-default.txt" 2>/dev/null || true

echo "Backup saved to $BACKUP_DIR"

2. Schedule Rollback (Mandatory for Remote)

The rollback restores from backup — not just disables the firewall — so Docker NAT and pre-existing rules are preserved. Dual-backend: at preferred, systemd-run fallback.

bash

# Build rollback script from backup dir
ROLLBACK_SCRIPT=$(cat <<'RB'
#!/bin/bash
BACKUP_DIR="REPLACE_ME"
[ -f "$BACKUP_DIR/iptables-v4.rules" ] && sudo iptables-restore < "$BACKUP_DIR/iptables-v4.rules" || { sudo iptables -P INPUT ACCEPT; sudo iptables -F; }
[ -f "$BACKUP_DIR/iptables-v6.rules" ] && sudo ip6tables-restore < "$BACKUP_DIR/iptables-v6.rules" || { sudo ip6tables -P INPUT ACCEPT; sudo ip6tables -F; }
[ -f "$BACKUP_DIR/nftables.rules" ] && sudo nft -f "$BACKUP_DIR/nftables.rules" || sudo nft flush ruleset
systemctl is-active ufw &>/dev/null && sudo ufw disable
sudo firewall-cmd --panic-off 2>/dev/null
RB
)
ROLLBACK_SCRIPT="${ROLLBACK_SCRIPT/REPLACE_ME/$BACKUP_DIR}"

# Schedule (at preferred, systemd-run fallback)
if command -v at &>/dev/null; then
    ROLLBACK_JOB_ID=$(echo "sudo bash -c '$ROLLBACK_SCRIPT'" | at now + 5 minutes 2>&1 | grep -oP 'job \K\d+')
    echo "Rollback scheduled: at job $ROLLBACK_JOB_ID (cancel with: atrm $ROLLBACK_JOB_ID)"
elif command -v systemd-run &>/dev/null; then
    UNIT_NAME="firewall-rollback-$$"
    echo "$ROLLBACK_SCRIPT" > /tmp/firewall-rollback-$$.sh
    chmod +x /tmp/firewall-rollback-$$.sh
    systemd-run --on-active=5m --unit="$UNIT_NAME" --user /tmp/firewall-rollback-$$.sh
    echo "Rollback scheduled: systemd unit $UNIT_NAME (cancel with: systemctl --user stop $UNIT_NAME)"
fi

See references/recovery.md for advanced recovery scenarios.

2a. Verify Rollback Timer Is Armed (Mandatory, Before Apply)

After scheduling the rollback but BEFORE applying new rules, confirm the timer is actually running:

bash

# For systemd-run
systemctl is-active --quiet firewall-rollback-$$.timer 2>/dev/null \
  && systemctl list-timers firewall-rollback-$$.timer --no-pager 2>/dev/null | grep -q firewall-rollback \
  || { echo "FATAL: rollback timer not scheduled — aborting before applying rules" >&2; exit 1; }

# For at
atq | grep -q "$ROLLBACK_JOB_ID" \
  || { echo "FATAL: rollback job $ROLLBACK_JOB_ID not found in queue — aborting" >&2; exit 1; }

Ordering constraint: This check runs AFTER scheduling but BEFORE apply. If the timer cannot be confirmed as armed, the script exits with code 31 and no rule change ever happens.

3. Pre-Flight Checklist

• Backup created successfully
• Rollback scheduled (verify with atq or systemctl --user list-units)
• Second SSH session open and tested — open a second terminal, SSH in, and confirm you can run sudo whoami. This is your emergency console if the primary session loses connectivity. Keep it open until VERIFY passes. Why: existing ESTABLISHED conntrack entries usually keep your current session alive, but if conntrack is flushed or the policy change drops your session silently, this second session is your only way back in.
• Real SSH port identified (not assumed to be 22)
• Confidence ≥ 70% and risk_tier is auto or confirmed
• Ownership verified — no IaC managing firewall
• Change window appropriate (maintenance window or low traffic)
• PLAN output reviewed and approved

---

State: APPLY

Apply firewall rules using the approved plan from PLAN state.

bash

# Get the approval_token from firewall-plan.sh --json output
bash scripts/firewall-apply.sh --approved-plan=sha256:abc123...
bash scripts/firewall-apply.sh --approved-plan=sha256:abc123... --dry-run

Apply behavior:

• Verifies approval_token matches current plan (exit 41 on mismatch — plan changed since approval)
• Checks for active rollback timer (exit 40 if VALIDATE was skipped)
• Supports all backends: ufw, firewalld, nftables, iptables
• Automatically runs firewall-verify.sh after applying
• Passes --dry-run for preview-only mode

Idempotent inline commands (for manual/scriptless use):

Backend	Pattern
ufw	sudo ufw status | awk '{print $1}' | grep -qx "22/tcp" || sudo ufw allow 22/tcp
firewalld	sudo firewall-cmd --query-service=ssh || sudo firewall-cmd --permanent --add-service=ssh
iptables	sudo iptables -C INPUT -p tcp --dport 22 -j ACCEPT 2>/dev/null || sudo iptables -A ...
nftables	Atomic ruleset: nft -c -f /etc/nftables.conf.new && nft -f /etc/nftables.conf.new

Docker Hosts

Docker bypasses ufw by default. Use DOCKER-USER chain. Full guide: references/docker-hardening.md.

Kubernetes Nodes

Default: AUDIT-ONLY. Never modify host firewall. Full policy: references/k8s-policy.md.

---

State: VERIFY

Phase 1 — Host-local verification (scripts/firewall-verify.sh):

bash

bash scripts/firewall-verify.sh

Phase 2 — External reachability check (scripts/container-port-audit.sh):

Requirement: All connectivity verification steps MUST be executed from a second, external host with an independent network path. Verifying from the target itself cannot detect a lockout.

From a second host on the same network segment, run:

bash

bash scripts/container-port-audit.sh <target-ip> <target-ip6>

This uses nc + /dev/tcp (not nmap) to verify expected-open ports are reachable and expected-closed canary ports are filtered, for both IPv4 and IPv6. UDP probes use nc -uzvw3; note that a "pass" may be a false-positive due to UDP being connectionless (a silently-dropping firewall produces the same result as an open port). Use local ss -ulnp as the deterministic anchor for UDP state.

If a second host is unavailable, the VERIFY stage also runs the container port audit and IPv4/IPv6 consistency checks locally:

bash

bash scripts/container-port-audit.sh   # local-only: container DNAT detection
bash scripts/ip-consistency.sh          # compare iptables vs ip6tables

Phase 3 — Container port DNAT audit:

Detects ports published by container runtimes that bypass the host's INPUT chain. Checks ss listeners + iptables -t nat DNAT rules + FORWARD chain entries. Flags any published port not reflected in explicit INPUT rules.

Success criteria (all must pass):

1. SSH remains reachable from current and second session
2. Only intended ports are externally reachable (verified from second host if available; otherwise local audit)
3. Rules survive reboot (verified via service persistence)
4. IPv6 exposure matches IPv4 policy (verified by ip-consistency.sh)
5. Docker-published ports are intentional (verified by container-port-audit.sh, no accidental 0.0.0.0)
6. fail2ban jails active (if installed) with correct backend
7. Rollback timer cancelled after successful verification

Verify behavior contract:

• Verify MUST complete within the rollback timer window (default 5 min)
• If verify times out before completion → timer auto-fires rollback (system-level protection)
• If verify FAILS but timer was already cancelled → manual rollback from the backup directory. Restore commands (in priority order):
  1. sudo iptables-restore < "$BACKUP_DIR/iptables-v4.rules"
  2. sudo ip6tables-restore < "$BACKUP_DIR/iptables-v6.rules"
  3. sudo nft -f "$BACKUP_DIR/nftables.rules"
  4. sudo ufw reset && sudo ufw disable See references/recovery.md for full recovery procedures including emergency ACCEPT fallback.
• The rollback is triggered by the timer (systemd-run/at), NOT by verify.sh itself — verify.sh exits with code 60 to signal failure, and the calling agent/scheduler handles the rollback decision

Exit Codes (Core Contract)

Code	Meaning	Agent Action
0	Success	Continue
10	Backend conflict	Halt; resolve manually
11	Backend detection failed	Halt; check firewall stack
12	Multiple backends active	Halt; resolve conflict
20	IaC-managed	Halt; update IaC source
21	Inside container	Halt; escalate to host operator
22	K8s node detected	Halt; audit-only mode
30	Low confidence (<70%)	Drop to audit-only mode
31	No rollback capability	Halt; ensure at or systemd-run
40	Preflight failed	Halt; check prerequisites
41	Plan approval mismatch	Halt; re-run PLAN with approval
42	RESERVED (Backup failed)	Halt; resolve disk/permissions
50	RESERVED (Apply failed)	Auto-rollback triggered
51	Apply partial	Auto-rollback triggered; verify backup
60	Verify failed	Auto-rollback triggered
61	RESERVED (State file conflict)	Abort; resolve stale state

---

fail2ban Integration

If fail2ban is installed:

Host Firewall	Recommended backend
ufw	ufw or systemd
firewalld	firewalld
nftables	nftables
iptables	auto (default)

After changing backend: sudo fail2ban-client restart && sudo fail2ban-client status sshd.

---

Recovery

If you lose connectivity, priority order:

1. Wait for auto-rollback (scheduled during VALIDATE)
2. Use second SSH session
3. Cloud serial console / hypervisor console
4. Restore from backup
5. Emergency ACCEPT (last resort — exposes host completely)

Full procedures: references/recovery.md.

State Persistence & Interrupt-Resume

For agent interrupt-resume scenarios (e.g., Apply failed mid-run, agent restarted), the state machine writes a lightweight state file to enable recovery without starting from Detect:

bash

STATE_DIR="$HOME/.firewall-hardening"
STATE_FILE="$STATE_DIR/state.json"

State file structure:

json

{
  "state": "validate",
  "started_at": "2026-05-11T16:00:00Z",
  "backend": "ufw",
  "risk_tier": "auto",
  "backup_dir": "/home/user/firewall-backup-20260511-160000",
  "rollback_timer_id": "firewall-rollback-12345",
  "plan_hash": "sha256:abc123..."
}

Resume logic:

• If state.json exists and started_at is within 1 hour → resume from that state
• If state.json is stale (>1 hour) → delete it and start fresh from Detect
• The file is advisory-only; agent can always restart from Detect

State persistence is optional. The skill defaults to restarting from Detect each run. Enable by creating $STATE_DIR before starting.

---

Cloud Security Group Reminder

The host firewall is your second layer. Verify cloud SGs are aligned:

Cloud	Outer Firewall
AWS	Security Groups
GCP	VPC Firewall Rules
Azure	Network Security Groups
DigitalOcean/Linode/Vultr	Cloud Firewall

Compatibility Matrix

Distro/Env	ufw	firewalld	nftables	iptables	Coverage
Ubuntu 22.04/24.04	Primary	—	Backend	Fallback	Full
Debian 12	Primary	—	Backend	Fallback	Full
RHEL 9	—	Primary	Native	Backend	Full
Rocky/Alma 9	—	Primary	Native	Backend	Full
Fedora 40+	—	Primary	Native	Backend	Partial
Alpine 3.18+	—	—	Native	Fallback	Partial
Arch	—	—	Native	Fallback	Community
Docker host	✅ DOCKER-USER chain	✅ docker-hardening.md	✅ docker-hardening.md	✅ docker-hardening.md	Full
LXC/LXD container	⚠️ Limited	⚠️ Limited	⚠️ Limited	⚠️ Limited	Partial
systemd-nspawn	⚠️ Limited	⚠️ Limited	⚠️ Limited	⚠️ Limited	Partial
WSL2	❌ Not supported	❌ Not supported	❌ Not supported	❌ Not supported	None

Container environments: Docker host is fully supported via DOCKER-USER chain. LXC/LXD/systemd-nspawn have limited support (kernel shares netfilter with host). WSL2 is explicitly unsupported. See references/special-environments.md.

---

Observability

Establish baselines after hardening: conntrack usage, dropped packet rates, fail2ban ban rate. Monitor for anomalies. Full guide: references/observability.md.

Compliance

Practices map to CIS, PCI-DSS, and SOC2 controls. Full mapping: references/compliance.md.

Quick Reference

Task	Command
Audit environment	bash scripts/audit-firewall.sh --json
Plan changes	bash scripts/firewall-plan.sh --profile web
Verify after apply	bash scripts/firewall-verify.sh
Allow port (ufw, idempotent)	sudo ufw status | awk '{print $1}' | grep -qx "80/tcp" || sudo ufw allow 80/tcp
View ufw rules	sudo ufw status numbered
View nft rules	sudo nft list ruleset
View iptables rules	sudo iptables -L -n -v
View ip6tables rules	sudo ip6tables -L -n -v
Atomic iptables replace	sudo iptables-restore < /tmp/rules.v4
Dry-run nftables	sudo nft -c -f /etc/nftables.conf
Backup rules	sudo iptables-save > ~/iptables.backup
fail2ban status	sudo fail2ban-client status sshd
Cancel rollback (at)	atrm <jobid>
Cancel rollback (systemd-run)	systemctl --user stop firewall-rollback-<pid>

See Also

• references/backend-ufw.md — Full UFW phase
• references/backend-firewalld.md — Full firewalld phase
• references/backend-nftables.md — Full nftables phase
• references/backend-iptables.md — Full iptables phase
• references/docker-hardening.md — Docker firewall hardening
• references/k8s-policy.md — Kubernetes node policy
• references/security-profiles.md — Pre-built configurations
• references/declarative-policy.md — YAML policy schema
• references/observability.md — Monitoring and baselines
• references/compliance.md — CIS/PCI-DSS/SOC2 mapping
• references/recovery.md — Recovery procedures
• references/special-environments.md — WSL2, containers, exit codes
• scripts/audit-firewall.sh — Environment detection
• scripts/firewall-plan.sh — Dry-run diff
• scripts/firewall-verify.sh — Post-apply verification