Linux Incident Remediator

Host-based Linux incident response and remediation skill focused on precise threat detection, forensic-safe data collection, firewall control (iptables/nftables), integrity validation, and controlled remediation while preserving system stability.

Linux Threat Mitigation and Incident Remediation (Hardened Edition)

This skill provides a structured, forensically-aware framework for analyzing and securing a Linux host during or after a security event.

It emphasizes:

• Non-destructive evidence collection
• Accurate threat detection
• Firewall-aware containment
• Integrity verification
• Controlled, reversible remediation
• Distribution-aware command usage

---

Environment Context

Supported Systems

• Debian / Ubuntu
• RHEL / CentOS / Rocky / Alma
• Fedora
• Arch Linux (limited package guidance)

Execution Assumptions

• Shell: bash or POSIX sh
• Privilege: Root or sudo
• Host-level access (NOT container-restricted environments)
• systemd-based systems preferred

⚠️ If running inside Docker, Kubernetes, LXC, or other containers, firewall, audit, and service commands may not reflect the host system.

---

Firewall Architecture Awareness

Modern Linux systems may use:

• iptables-legacy
• iptables-nft (compatibility wrapper)
• Native nftables
• firewalld (RHEL-family default)

Identify Firewall Backend

bash

iptables --version
which nft
systemctl status firewalld

If nftables is active:

bash

nft list ruleset

Do NOT assume iptables -L represents the full firewall state.

---

Logging Differences by Distribution

Distribution	Primary Log File
Ubuntu/Debian	/var/log/syslog
RHEL/CentOS/Fedora	/var/log/messages
All modern systemd	journalctl

Always prefer:

bash

journalctl -xe

---

Operational Toolkit (Hardened)

1. Network Inspection

Listening Services

bash

ss -tulpn

Active Connections

bash

ss -antp | grep ESTABLISHED

Firewall State

iptables

bash

iptables -L -n -v --line-numbers
iptables -S

nftables

bash

nft list ruleset

Local Service Enumeration (Low Noise)

bash

ss -lntup

Avoid unnecessary full scans of localhost unless required.

Conservative Network Scan

bash

nmap -sV -T3 -p- localhost

Packet Capture (Short Snapshot)

bash

tcpdump -i any -nn -c 100

---

2. Process & Runtime Analysis

Process Tree

bash

ps auxww --forest

High CPU / Memory

bash

top

Open File Handles

bash

lsof -p <PID>

System Call Trace (Caution: Alters Timing)

bash

strace -p <PID>

⚠️ strace may change process behavior. Use carefully during live compromise.

Kernel Modules

bash

lsmod

Kernel Messages

bash

dmesg | tail -50

---

3. Rootkit & Malware Scanning

Rootkit Scanners

bash

rkhunter --check
chkrootkit

May produce false positives. Validate findings manually.

Antivirus Scan (Targeted)

bash

clamscan -r /home

Use selectively; large scans increase I/O and may alter access timestamps.

Lynis System Audit

bash

lynis audit system

---

4. File Integrity & Package Verification

AIDE (After Initialization)

Install:

bash

apt install aide
# or
dnf install aide

Initialize:

bash

aideinit
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Run Check:

bash

aide --check

RHEL Package Verification

bash

rpm -Va

Debian Package Verification

bash

apt install debsums
debsums -s

---

5. Forensic Analysis (Didier Stevens Suite)

Install:

bash

sudo mkdir -p /opt/forensics
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/base64dump.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/re-search.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/zipdump.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/1768.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/pdf-parser.py
sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/oledump.py
sudo chmod +x /opt/forensics/*.py

Decode Base64

bash

python3 /opt/forensics/base64dump.py file.txt

IOC Search

bash

python3 /opt/forensics/re-search.py -n ipv4 logfile

Inspect ZIP (No Extraction)

bash

python3 /opt/forensics/zipdump.py suspicious.zip

Extract Cobalt Strike Beacon Config

bash

python3 /opt/forensics/1768.py payload.bin

Inspect Office/PDF Documents

bash

python3 /opt/forensics/pdf-parser.py file.pdf
python3 /opt/forensics/oledump.py file.doc

Static inspection only. Never execute suspicious files.

---

6. Authentication & User Activity

Current Sessions

bash

who -a

Login History

bash

last -a

Failed SSH Logins

Ubuntu/Debian:

bash

journalctl -u ssh.service | grep "Failed password"

RHEL/Fedora:

bash

journalctl -u sshd.service | grep "Failed password"

Sudo Activity

bash

journalctl _COMM=sudo

Audit Logs

bash

ausearch -m USER_AUTH,USER_LOGIN,USER_CHAUTHTOK

---

Controlled Remediation

Blocking an IP

iptables (Immediate)

bash

iptables -I INPUT 1 -s <IP> -j DROP

nftables

bash

nft add rule inet filter input ip saddr <IP> drop

If firewalld is active:

bash

firewall-cmd --add-rich-rule='rule family="ipv4" source address="<IP>" drop'

---

Persisting Firewall Rules

iptables (Debian):

bash

netfilter-persistent save

iptables (manual save):

bash

iptables-save > /etc/iptables/rules.v4

firewalld:

bash

firewall-cmd --runtime-to-permanent

nftables:

bash

nft list ruleset > /etc/nftables.conf

---

Process Containment Strategy

Preferred escalation:

1. Observe
2. kill -TERM <PID>
3. If required: kill -STOP <PID> for analysis
4. Use kill -KILL <PID> only if necessary

Avoid killall or broad pkill.

---

Service Isolation

bash

systemctl stop <service>
systemctl disable <service>
systemctl mask <service>

---

Persistence & Backdoor Checks

Cron Jobs

bash

crontab -l
ls -lah /etc/cron*

Systemd Persistence

bash

ls -lah /etc/systemd/system/

Startup Scripts

bash

cat /etc/rc.local

---

SELinux Awareness (RHEL/Fedora)

Check status:

bash

getenforce

Review denials:

bash

ausearch -m AVC

---

Forensic Hygiene

1. Never execute suspicious binaries.
2. Preserve evidence before deletion:

bash

sha256sum file
mkdir -p /root/quarantine
mv file /root/quarantine/file.vir

3. Log every remediation step:

bash

date -u

Document:

• Timestamp
• Command executed
• Observed outcome

---

Usage Examples

Routine Audit

• Run lynis audit system
• Verify no unknown listening services
• Check for modified system binaries

Active Threat

• Identify high CPU process
• Capture short tcpdump
• Extract file hash
• Contain IP via firewall
• Preserve malicious artifact

Suspicious File

• Use zipdump
• Extract hash
• Move to quarantine
• Search logs for execution attempts

---

Safety Guardrails

These guardrails are mandatory and apply to all remediation activity. Their purpose is to prevent self-inflicted outages, preserve forensic integrity, and ensure reversible, controlled incident response.

---

1. State Verification (Pre- and Post-Change Validation)

Before executing any remediation command:

1. Record timestamp (UTC):

   bash

   date -u
   

2. Run a discovery command to capture current state:

   • Network: ss -tulpn
   • Active connections: ss -antp
   • Firewall (iptables): iptables -L -n -v
   • Firewall (nftables): nft list ruleset
   • firewalld: firewall-cmd --list-all

After remediation:

3. Re-run the same discovery command.
4. Compare state change and confirm:
   • Intended effect achieved
   • No unintended service disruption
   • No management lockout (e.g., SSH access intact)

Never assume a command succeeded without verifying its effect.

---

2. No Wildcards or Broad Termination

To prevent catastrophic system damage:

• NEVER use:

  • rm -rf *
  • rm -rf /
  • killall
  • Broad pkill patterns
  • Unbounded globbing in sensitive directories

• Always:

  • Use absolute file paths (e.g., /tmp/malware.bin)
  • Target explicit PIDs (kill -TERM <PID>)
  • Confirm file existence with ls -lah <file>
  • Hash suspicious files before modification:

    bash

    sha256sum <file>
    

Wildcard deletions and pattern-based termination are prohibited during incident response.

---

3. Persistence & Re-Spawn Inspection

After containment of a malicious process or service, immediately inspect for persistence mechanisms.

Check:

Cron Jobs

bash

crontab -l
ls -lah /etc/cron*

systemd Services & Timers

bash

systemctl list-unit-files --type=service
systemctl list-timers --all
ls -lah /etc/systemd/system/

Init Scripts

bash

ls -lah /etc/init.d/
cat /etc/rc.local

User-Level Persistence

bash

ls -lah ~/.config/systemd/user/

SSH Backdoors

bash

cat ~/.ssh/authorized_keys

After removal of malicious artifacts:

• Run integrity verification:

  bash

  aide --check
  

• On RHEL-based systems:

  bash

  rpm -Va
  

• On Debian-based systems:

  bash

  debsums -s
  

Do not consider a threat eradicated until persistence mechanisms are eliminated.

---

4. Firewall Rule Safety & Persistence

A. Anti-Lockout Requirement

Before modifying firewall rules:

1. Confirm SSH listening port:

   bash

   ss -tulpn | grep ssh
   

2. Confirm an explicit ACCEPT rule exists for:

   • Current management IP
   • SSH port

NEVER:

bash

iptables -F

NEVER set a default DROP policy without verifying SSH access rule exists.

---

B. Immediate vs Persistent Rules

Firewall rule changes are runtime by default and may not survive reboot.

iptables (Debian/Ubuntu)

Runtime only until saved:

bash

iptables-save > /etc/iptables/rules.v4

If using netfilter-persistent:

bash

netfilter-persistent save

RHEL (legacy iptables service)

bash

service iptables save

firewalld

Runtime-to-permanent:

bash

firewall-cmd --runtime-to-permanent

nftables

Persist ruleset:

bash

nft list ruleset > /etc/nftables.conf

Document:

• Whether rule is temporary or permanent
• Location of saved configuration
• Verification after reboot (if applicable)

---

5. Forensic Preservation Before Destruction

Before deleting or killing:

1. Hash the artifact:

   bash

   sha256sum <file>
   

2. Move to quarantine:

   bash

   mkdir -p /root/quarantine
   mv <file> /root/quarantine/<file>.vir
   

3. Record:

   • Timestamp (UTC)
   • Original path
   • Hash value
   • Reason for containment

Avoid kill -9 unless absolutely required. Prefer:

1. kill -TERM <PID>
2. kill -STOP <PID> (if forensic inspection needed)
3. kill -KILL <PID> only as last resort

---

6. Change Logging Requirement

Every remediation action must include:

• date -u
• Command executed
• Justification
• Observed outcome
• Updated risk level (if applicable)

Remediation without documentation is non-compliant.

---

7. Minimal-Impact Principle

All actions must follow:

• Smallest necessary change
• Reversible where possible
• No broad configuration resets
• No service restarts without justification
• No system-wide scans during active compromise unless scoped

Contain first. Eradicate methodically. Recover cautiously.