{"skill":{"slug":"sys-guard-linux-remediator","displayName":"Linux Incident Remediator","summary":"Provides forensically-safe Linux threat detection, network and process analysis, integrity verification, controlled firewall and service remediation preservi...","description":"---\nname: sys-guard-linux-remediator\ndescription: Host-based Linux incident response and remediation skill focused on precise threat detection, forensic-safe data collection, firewall control (iptables/nftables), integrity validation, and controlled remediation while preserving system stability.\nmetadata:\n  author: Edwin Kairu (ekairu@cmu.edu)\n---\n\n# Linux Threat Mitigation and Incident Remediation (Hardened Edition)\n\nThis skill provides a structured, forensically-aware framework for analyzing and securing a Linux host during or after a security event.\n\nIt emphasizes:\n\n- Non-destructive evidence collection\n- Accurate threat detection\n- Firewall-aware containment\n- Integrity verification\n- Controlled, reversible remediation\n- Distribution-aware command usage\n\n---\n\n# Environment Context\n\n## Supported Systems\n\n- Debian / Ubuntu\n- RHEL / CentOS / Rocky / Alma\n- Fedora\n- Arch Linux (limited package guidance)\n\n## Execution Assumptions\n\n- Shell: `bash` or POSIX `sh`\n- Privilege: Root or sudo\n- Host-level access (NOT container-restricted environments)\n- systemd-based systems preferred\n\n> ⚠️ If running inside Docker, Kubernetes, LXC, or other containers, firewall, audit, and service commands may not reflect the host system.\n\n---\n\n# Firewall Architecture Awareness\n\nModern Linux systems may use:\n\n- `iptables-legacy`\n- `iptables-nft` (compatibility wrapper)\n- Native `nftables`\n- `firewalld` (RHEL-family default)\n\n## Identify Firewall Backend\n\n```bash\niptables --version\nwhich nft\nsystemctl status firewalld\n```\n\nIf nftables is active:\n\n```bash\nnft list ruleset\n```\n\nDo NOT assume `iptables -L` represents the full firewall state.\n\n---\n\n# Logging Differences by Distribution\n\n| Distribution | Primary Log File |\n|--------------|------------------|\n| Ubuntu/Debian | `/var/log/syslog` |\n| RHEL/CentOS/Fedora | `/var/log/messages` |\n| All modern systemd | `journalctl` |\n\nAlways prefer:\n\n```bash\njournalctl -xe\n```\n\n---\n\n# Operational Toolkit (Hardened)\n\n## 1. Network Inspection\n\n### Listening Services\n```bash\nss -tulpn\n```\n\n### Active Connections\n```bash\nss -antp | grep ESTABLISHED\n```\n\n### Firewall State\n\n#### iptables\n```bash\niptables -L -n -v --line-numbers\niptables -S\n```\n\n#### nftables\n```bash\nnft list ruleset\n```\n\n### Local Service Enumeration (Low Noise)\n```bash\nss -lntup\n```\n\nAvoid unnecessary full scans of localhost unless required.\n\n### Conservative Network Scan\n```bash\nnmap -sV -T3 -p- localhost\n```\n\n### Packet Capture (Short Snapshot)\n```bash\ntcpdump -i any -nn -c 100\n```\n\n---\n\n## 2. Process & Runtime Analysis\n\n### Process Tree\n```bash\nps auxww --forest\n```\n\n### High CPU / Memory\n```bash\ntop\n```\n\n### Open File Handles\n```bash\nlsof -p <PID>\n```\n\n### System Call Trace (Caution: Alters Timing)\n```bash\nstrace -p <PID>\n```\n\n> ⚠️ `strace` may change process behavior. Use carefully during live compromise.\n\n### Kernel Modules\n```bash\nlsmod\n```\n\n### Kernel Messages\n```bash\ndmesg | tail -50\n```\n\n---\n\n## 3. Rootkit & Malware Scanning\n\n### Rootkit Scanners\n```bash\nrkhunter --check\nchkrootkit\n```\n\n> May produce false positives. Validate findings manually.\n\n### Antivirus Scan (Targeted)\n```bash\nclamscan -r /home\n```\n\nUse selectively; large scans increase I/O and may alter access timestamps.\n\n### Lynis System Audit\n```bash\nlynis audit system\n```\n\n---\n\n## 4. File Integrity & Package Verification\n\n### AIDE (After Initialization)\n\nInstall:\n```bash\napt install aide\n# or\ndnf install aide\n```\n\nInitialize:\n```bash\naideinit\nmv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz\n```\n\nRun Check:\n```bash\naide --check\n```\n\n### RHEL Package Verification\n```bash\nrpm -Va\n```\n\n### Debian Package Verification\n```bash\napt install debsums\ndebsums -s\n```\n\n---\n\n## 5. Forensic Analysis (Didier Stevens Suite)\n\nInstall:\n\n```bash\nsudo mkdir -p /opt/forensics\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/base64dump.py\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/re-search.py\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/zipdump.py\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/1768.py\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/pdf-parser.py\nsudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/oledump.py\nsudo chmod +x /opt/forensics/*.py\n```\n\n### Decode Base64\n```bash\npython3 /opt/forensics/base64dump.py file.txt\n```\n\n### IOC Search\n```bash\npython3 /opt/forensics/re-search.py -n ipv4 logfile\n```\n\n### Inspect ZIP (No Extraction)\n```bash\npython3 /opt/forensics/zipdump.py suspicious.zip\n```\n\n### Extract Cobalt Strike Beacon Config\n```bash\npython3 /opt/forensics/1768.py payload.bin\n```\n\n### Inspect Office/PDF Documents\n```bash\npython3 /opt/forensics/pdf-parser.py file.pdf\npython3 /opt/forensics/oledump.py file.doc\n```\n\n> Static inspection only. Never execute suspicious files.\n\n---\n\n## 6. Authentication & User Activity\n\n### Current Sessions\n```bash\nwho -a\n```\n\n### Login History\n```bash\nlast -a\n```\n\n### Failed SSH Logins\n\nUbuntu/Debian:\n```bash\njournalctl -u ssh.service | grep \"Failed password\"\n```\n\nRHEL/Fedora:\n```bash\njournalctl -u sshd.service | grep \"Failed password\"\n```\n\n### Sudo Activity\n```bash\njournalctl _COMM=sudo\n```\n\n### Audit Logs\n```bash\nausearch -m USER_AUTH,USER_LOGIN,USER_CHAUTHTOK\n```\n\n---\n\n# Controlled Remediation\n\n## Blocking an IP\n\n### iptables (Immediate)\n```bash\niptables -I INPUT 1 -s <IP> -j DROP\n```\n\n### nftables\n```bash\nnft add rule inet filter input ip saddr <IP> drop\n```\n\nIf firewalld is active:\n```bash\nfirewall-cmd --add-rich-rule='rule family=\"ipv4\" source address=\"<IP>\" drop'\n```\n\n---\n\n## Persisting Firewall Rules\n\niptables (Debian):\n```bash\nnetfilter-persistent save\n```\n\niptables (manual save):\n```bash\niptables-save > /etc/iptables/rules.v4\n```\n\nfirewalld:\n```bash\nfirewall-cmd --runtime-to-permanent\n```\n\nnftables:\n```bash\nnft list ruleset > /etc/nftables.conf\n```\n\n---\n\n## Process Containment Strategy\n\nPreferred escalation:\n\n1. Observe\n2. `kill -TERM <PID>`\n3. If required: `kill -STOP <PID>` for analysis\n4. Use `kill -KILL <PID>` only if necessary\n\nAvoid `killall` or broad `pkill`.\n\n---\n\n## Service Isolation\n\n```bash\nsystemctl stop <service>\nsystemctl disable <service>\nsystemctl mask <service>\n```\n\n---\n\n# Persistence & Backdoor Checks\n\n### Cron Jobs\n```bash\ncrontab -l\nls -lah /etc/cron*\n```\n\n### Systemd Persistence\n```bash\nls -lah /etc/systemd/system/\n```\n\n### Startup Scripts\n```bash\ncat /etc/rc.local\n```\n\n---\n\n# SELinux Awareness (RHEL/Fedora)\n\nCheck status:\n```bash\ngetenforce\n```\n\nReview denials:\n```bash\nausearch -m AVC\n```\n\n---\n\n# Forensic Hygiene\n\n1. Never execute suspicious binaries.\n2. Preserve evidence before deletion:\n\n```bash\nsha256sum file\nmkdir -p /root/quarantine\nmv file /root/quarantine/file.vir\n```\n\n3. Log every remediation step:\n\n```bash\ndate -u\n```\n\nDocument:\n- Timestamp\n- Command executed\n- Observed outcome\n\n---\n\n# Usage Examples\n\n## Routine Audit\n\n- Run `lynis audit system`\n- Verify no unknown listening services\n- Check for modified system binaries\n\n## Active Threat\n\n- Identify high CPU process\n- Capture short `tcpdump`\n- Extract file hash\n- Contain IP via firewall\n- Preserve malicious artifact\n\n## Suspicious File\n\n- Use `zipdump`\n- Extract hash\n- Move to quarantine\n- Search logs for execution attempts\n\n---\n\n# Safety Guardrails\n\nThese guardrails are mandatory and apply to all remediation activity. Their purpose is to prevent self-inflicted outages, preserve forensic integrity, and ensure reversible, controlled incident response.\n\n---\n\n## 1. State Verification (Pre- and Post-Change Validation)\n\nBefore executing any remediation command:\n\n1. Record timestamp (UTC):\n   ```bash\n   date -u\n   ```\n\n2. Run a discovery command to capture current state:\n   - Network: `ss -tulpn`\n   - Active connections: `ss -antp`\n   - Firewall (iptables): `iptables -L -n -v`\n   - Firewall (nftables): `nft list ruleset`\n   - firewalld: `firewall-cmd --list-all`\n\nAfter remediation:\n\n3. Re-run the same discovery command.\n4. Compare state change and confirm:\n   - Intended effect achieved\n   - No unintended service disruption\n   - No management lockout (e.g., SSH access intact)\n\nNever assume a command succeeded without verifying its effect.\n\n---\n\n## 2. No Wildcards or Broad Termination\n\nTo prevent catastrophic system damage:\n\n- NEVER use:\n  - `rm -rf *`\n  - `rm -rf /`\n  - `killall`\n  - Broad `pkill` patterns\n  - Unbounded globbing in sensitive directories\n\n- Always:\n  - Use absolute file paths (e.g., `/tmp/malware.bin`)\n  - Target explicit PIDs (`kill -TERM <PID>`)\n  - Confirm file existence with `ls -lah <file>`\n  - Hash suspicious files before modification:\n    ```bash\n    sha256sum <file>\n    ```\n\nWildcard deletions and pattern-based termination are prohibited during incident response.\n\n---\n\n## 3. Persistence & Re-Spawn Inspection\n\nAfter containment of a malicious process or service, immediately inspect for persistence mechanisms.\n\n### Check:\n\n#### Cron Jobs\n```bash\ncrontab -l\nls -lah /etc/cron*\n```\n\n#### systemd Services & Timers\n```bash\nsystemctl list-unit-files --type=service\nsystemctl list-timers --all\nls -lah /etc/systemd/system/\n```\n\n#### Init Scripts\n```bash\nls -lah /etc/init.d/\ncat /etc/rc.local\n```\n\n#### User-Level Persistence\n```bash\nls -lah ~/.config/systemd/user/\n```\n\n#### SSH Backdoors\n```bash\ncat ~/.ssh/authorized_keys\n```\n\nAfter removal of malicious artifacts:\n\n- Run integrity verification:\n  ```bash\n  aide --check\n  ```\n- On RHEL-based systems:\n  ```bash\n  rpm -Va\n  ```\n- On Debian-based systems:\n  ```bash\n  debsums -s\n  ```\n\nDo not consider a threat eradicated until persistence mechanisms are eliminated.\n\n---\n\n## 4. Firewall Rule Safety & Persistence\n\n### A. Anti-Lockout Requirement\n\nBefore modifying firewall rules:\n\n1. Confirm SSH listening port:\n   ```bash\n   ss -tulpn | grep ssh\n   ```\n\n2. Confirm an explicit ACCEPT rule exists for:\n   - Current management IP\n   - SSH port\n\nNEVER:\n```bash\niptables -F\n```\n\nNEVER set a default DROP policy without verifying SSH access rule exists.\n\n---\n\n### B. Immediate vs Persistent Rules\n\nFirewall rule changes are runtime by default and may not survive reboot.\n\n#### iptables (Debian/Ubuntu)\nRuntime only until saved:\n```bash\niptables-save > /etc/iptables/rules.v4\n```\n\nIf using netfilter-persistent:\n```bash\nnetfilter-persistent save\n```\n\n#### RHEL (legacy iptables service)\n```bash\nservice iptables save\n```\n\n#### firewalld\nRuntime-to-permanent:\n```bash\nfirewall-cmd --runtime-to-permanent\n```\n\n#### nftables\nPersist ruleset:\n```bash\nnft list ruleset > /etc/nftables.conf\n```\n\nDocument:\n- Whether rule is temporary or permanent\n- Location of saved configuration\n- Verification after reboot (if applicable)\n\n---\n\n## 5. Forensic Preservation Before Destruction\n\nBefore deleting or killing:\n\n1. Hash the artifact:\n   ```bash\n   sha256sum <file>\n   ```\n\n2. Move to quarantine:\n   ```bash\n   mkdir -p /root/quarantine\n   mv <file> /root/quarantine/<file>.vir\n   ```\n\n3. Record:\n   - Timestamp (UTC)\n   - Original path\n   - Hash value\n   - Reason for containment\n\nAvoid `kill -9` unless absolutely required. Prefer:\n\n1. `kill -TERM <PID>`\n2. `kill -STOP <PID>` (if forensic inspection needed)\n3. `kill -KILL <PID>` only as last resort\n\n---\n\n## 6. Change Logging Requirement\n\nEvery remediation action must include:\n\n- `date -u`\n- Command executed\n- Justification\n- Observed outcome\n- Updated risk level (if applicable)\n\nRemediation without documentation is non-compliant.\n\n---\n\n## 7. Minimal-Impact Principle\n\nAll actions must follow:\n\n- Smallest necessary change\n- Reversible where possible\n- No broad configuration resets\n- No service restarts without justification\n- No system-wide scans during active compromise unless scoped\n\nContain first. Eradicate methodically. Recover cautiously.\n\n","tags":{"latest":"1.0.0"},"stats":{"comments":0,"downloads":1312,"installsAllTime":49,"installsCurrent":0,"stars":0,"versions":1},"createdAt":1771060398923,"updatedAt":1778731374084},"latestVersion":{"version":"1.0.0","createdAt":1771060398923,"changelog":"Initial release of sys-guard-linux-remediator: a Linux incident response and remediation skill.\n\n- Forensic-safe threat detection and evidence collection with minimal system disruption\n- Comprehensive firewall detection and handling (iptables, nftables, firewalld)\n- Toolkit for live analysis: network, process, integrity, malware/risk, and user activity inspection\n- Structured, reversible remediation steps with operational safety guardrails\n- Clear guidance for distribution variations, forensic hygiene, and persistent threat checks","license":null},"metadata":null,"owner":{"handle":"kiaraho","userId":"s17dd7d5z6ddkv006q10r1sf8s885ntg","displayName":"kiaraho","image":"https://avatars.githubusercontent.com/u/5502596?v=4"},"moderation":null}