ClawHub

Linux Incident Remediator

Other

Host-based Linux incident response and remediation skill focused on precise threat detection, forensic-safe data collection, firewall control (iptables/nftables), integrity validation, and controlled remediation while preserving system stability.

Install

openclaw skills install sys-guard-linux-remediator

Linux Threat Mitigation and Incident Remediation (Hardened Edition)

This skill provides a structured, forensically-aware framework for analyzing and securing a Linux host during or after a security event.

It emphasizes:

  • Non-destructive evidence collection
  • Accurate threat detection
  • Firewall-aware containment
  • Integrity verification
  • Controlled, reversible remediation
  • Distribution-aware command usage

Environment Context

Supported Systems

  • Debian / Ubuntu
  • RHEL / CentOS / Rocky / Alma
  • Fedora
  • Arch Linux (limited package guidance)

Execution Assumptions

  • Shell: bash or POSIX sh
  • Privilege: Root or sudo
  • Host-level access (NOT container-restricted environments)
  • systemd-based systems preferred

⚠️ If running inside Docker, Kubernetes, LXC, or other containers, firewall, audit, and service commands may not reflect the host system.

Firewall Architecture Awareness

Modern Linux systems may use:

  • iptables-legacy
  • iptables-nft (compatibility wrapper)
  • Native nftables
  • firewalld (RHEL-family default)

Identify Firewall Backend

iptables --version
which nft
systemctl status firewalld

If nftables is active:

nft list ruleset

Do NOT assume iptables -L represents the full firewall state.

Logging Differences by Distribution

DistributionPrimary Log File
Ubuntu/Debian/var/log/syslog
RHEL/CentOS/Fedora/var/log/messages
All modern systemdjournalctl

Always prefer:

journalctl -xe

Operational Toolkit (Hardened)

1. Network Inspection

Listening Services

ss -tulpn

Active Connections

ss -antp | grep ESTABLISHED

Firewall State

iptables

iptables -L -n -v --line-numbers
iptables -S

nftables

nft list ruleset

Local Service Enumeration (Low Noise)

ss -lntup

Avoid unnecessary full scans of localhost unless required.

Conservative Network Scan

nmap -sV -T3 -p- localhost

Packet Capture (Short Snapshot)

tcpdump -i any -nn -c 100

2. Process & Runtime Analysis

Process Tree

ps auxww --forest

High CPU / Memory

top

Open File Handles

lsof -p <PID>

System Call Trace (Caution: Alters Timing)

strace -p <PID>

⚠️ strace may change process behavior. Use carefully during live compromise.

Kernel Modules

lsmod

Kernel Messages

dmesg | tail -50

3. Rootkit & Malware Scanning

Rootkit Scanners

rkhunter --check
chkrootkit

May produce false positives. Validate findings manually.

Antivirus Scan (Targeted)

clamscan -r /home

Use selectively; large scans increase I/O and may alter access timestamps.

Lynis System Audit

lynis audit system

4. File Integrity & Package Verification

AIDE (After Initialization)

Install:

apt install aide
# or
dnf install aide

Initialize:

aideinit
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Run Check:

aide --check

RHEL Package Verification

rpm -Va

Debian Package Verification

apt install debsums
debsums -s

5. Forensic Analysis (Didier Stevens Suite)

Install:

sudo mkdir -p /opt/forensics
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/base64dump.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/re-search.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/zipdump.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/1768.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/pdf-parser.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/oledump.py
sudo chmod +x /opt/forensics/*.py

Decode Base64

python3 /opt/forensics/base64dump.py file.txt

IOC Search

python3 /opt/forensics/re-search.py -n ipv4 logfile

Inspect ZIP (No Extraction)

python3 /opt/forensics/zipdump.py suspicious.zip

Extract Cobalt Strike Beacon Config

python3 /opt/forensics/1768.py payload.bin

Inspect Office/PDF Documents

python3 /opt/forensics/pdf-parser.py file.pdf
python3 /opt/forensics/oledump.py file.doc

Static inspection only. Never execute suspicious files.

6. Authentication & User Activity

Current Sessions

who -a

Login History

last -a

Failed SSH Logins

Ubuntu/Debian:

journalctl -u ssh.service | grep "Failed password"

RHEL/Fedora:

journalctl -u sshd.service | grep "Failed password"

Sudo Activity

journalctl _COMM=sudo

Audit Logs

ausearch -m USER_AUTH,USER_LOGIN,USER_CHAUTHTOK

Controlled Remediation

Blocking an IP

iptables (Immediate)

iptables -I INPUT 1 -s <IP> -j DROP

nftables

nft add rule inet filter input ip saddr <IP> drop

If firewalld is active:

firewall-cmd --add-rich-rule='rule family="ipv4" source address="<IP>" drop'

Persisting Firewall Rules

iptables (Debian):

netfilter-persistent save

iptables (manual save):

iptables-save > /etc/iptables/rules.v4

firewalld:

firewall-cmd --runtime-to-permanent

nftables:

nft list ruleset > /etc/nftables.conf

Process Containment Strategy

Preferred escalation:

  1. Observe
  2. kill -TERM <PID>
  3. If required: kill -STOP <PID> for analysis
  4. Use kill -KILL <PID> only if necessary

Avoid killall or broad pkill.

Service Isolation

systemctl stop <service>
systemctl disable <service>
systemctl mask <service>

Persistence & Backdoor Checks

Cron Jobs

crontab -l
ls -lah /etc/cron*

Systemd Persistence

ls -lah /etc/systemd/system/

Startup Scripts

cat /etc/rc.local

SELinux Awareness (RHEL/Fedora)

Check status:

getenforce

Review denials:

ausearch -m AVC

Forensic Hygiene

  1. Never execute suspicious binaries.
  2. Preserve evidence before deletion:
sha256sum file
mkdir -p /root/quarantine
mv file /root/quarantine/file.vir
  1. Log every remediation step:
date -u

Document:

  • Timestamp
  • Command executed
  • Observed outcome

Usage Examples

Routine Audit

  • Run lynis audit system
  • Verify no unknown listening services
  • Check for modified system binaries

Active Threat

  • Identify high CPU process
  • Capture short tcpdump
  • Extract file hash
  • Contain IP via firewall
  • Preserve malicious artifact

Suspicious File

  • Use zipdump
  • Extract hash
  • Move to quarantine
  • Search logs for execution attempts

Safety Guardrails

These guardrails are mandatory and apply to all remediation activity. Their purpose is to prevent self-inflicted outages, preserve forensic integrity, and ensure reversible, controlled incident response.

1. State Verification (Pre- and Post-Change Validation)

Before executing any remediation command:

  1. Record timestamp (UTC):

    date -u

  2. Run a discovery command to capture current state:

    • Network: ss -tulpn
    • Active connections: ss -antp
    • Firewall (iptables): iptables -L -n -v
    • Firewall (nftables): nft list ruleset
    • firewalld: firewall-cmd --list-all

After remediation:

  1. Re-run the same discovery command.
  2. Compare state change and confirm:
    • Intended effect achieved
    • No unintended service disruption
    • No management lockout (e.g., SSH access intact)

Never assume a command succeeded without verifying its effect.

2. No Wildcards or Broad Termination

To prevent catastrophic system damage:

  • NEVER use:

    • rm -rf *
    • rm -rf /
    • killall
    • Broad pkill patterns
    • Unbounded globbing in sensitive directories

  • Always:

    • Use absolute file paths (e.g., /tmp/malware.bin)
    • Target explicit PIDs (kill -TERM <PID>)
    • Confirm file existence with ls -lah <file>
    • Hash suspicious files before modification: 
      sha256sum <file>

Wildcard deletions and pattern-based termination are prohibited during incident response.

3. Persistence & Re-Spawn Inspection

After containment of a malicious process or service, immediately inspect for persistence mechanisms.

Check:

Cron Jobs

crontab -l
ls -lah /etc/cron*

systemd Services & Timers

systemctl list-unit-files --type=service
systemctl list-timers --all
ls -lah /etc/systemd/system/

Init Scripts

ls -lah /etc/init.d/
cat /etc/rc.local

User-Level Persistence

ls -lah ~/.config/systemd/user/

SSH Backdoors

cat ~/.ssh/authorized_keys

After removal of malicious artifacts:

  • Run integrity verification: 
    aide --check
  • On RHEL-based systems: 
    rpm -Va
  • On Debian-based systems: 
    debsums -s

Do not consider a threat eradicated until persistence mechanisms are eliminated.

4. Firewall Rule Safety & Persistence

A. Anti-Lockout Requirement

Before modifying firewall rules:

  1. Confirm SSH listening port:

    ss -tulpn | grep ssh

  2. Confirm an explicit ACCEPT rule exists for:

    • Current management IP
    • SSH port

NEVER:

iptables -F

NEVER set a default DROP policy without verifying SSH access rule exists.

B. Immediate vs Persistent Rules

Firewall rule changes are runtime by default and may not survive reboot.

iptables (Debian/Ubuntu)

Runtime only until saved:

iptables-save > /etc/iptables/rules.v4

If using netfilter-persistent:

netfilter-persistent save

RHEL (legacy iptables service)

service iptables save

firewalld

Runtime-to-permanent:

firewall-cmd --runtime-to-permanent

nftables

Persist ruleset:

nft list ruleset > /etc/nftables.conf

Document:

  • Whether rule is temporary or permanent
  • Location of saved configuration
  • Verification after reboot (if applicable)

5. Forensic Preservation Before Destruction

Before deleting or killing:

  1. Hash the artifact:

    sha256sum <file>

  2. Move to quarantine:

    mkdir -p /root/quarantine
mv <file> /root/quarantine/<file>.vir

  3. Record:

    • Timestamp (UTC)
    • Original path
    • Hash value
    • Reason for containment

Avoid kill -9 unless absolutely required. Prefer:

  1. kill -TERM <PID>
  2. kill -STOP <PID> (if forensic inspection needed)
  3. kill -KILL <PID> only as last resort

6. Change Logging Requirement

Every remediation action must include:

  • date -u
  • Command executed
  • Justification
  • Observed outcome
  • Updated risk level (if applicable)

Remediation without documentation is non-compliant.

7. Minimal-Impact Principle

All actions must follow:

  • Smallest necessary change
  • Reversible where possible
  • No broad configuration resets
  • No service restarts without justification
  • No system-wide scans during active compromise unless scoped

Contain first. Eradicate methodically. Recover cautiously.