Linux

Debugs and hardens Linux hosts: permissions, disk full, OOM kills, stuck processes, systemd units, cron, networking, SSH, and boot failures. Use when a service starts by hand but fails at boot, a process ignores kill -9, df and du disagree, a box runs out of memory or inodes, sudo or ACLs deny access, SELinux blocks a write, a job works in the shell but not in cron, sshd rejects a key, an upgrade leaves packages half-configured, load is high while the CPU sits idle, or a host needs firewall rules, users, LVM, journald, kernel tuning, or a security baseline. Also for setting up a fresh server, deciding what to alert on, backups whose restore has never been tested, a host that may be compromised, and desktop or laptop trouble — GPU drivers, Wayland, suspend, audio, Wi-Fi. Covers Debian/Ubuntu, RHEL/Fedora, Arch, Alpine, SUSE and WSL. Not for shell-script syntax (bash) or container build and runtime internals (docker).

Data. At the start of every session, read ~/Clawic/data/linux/config.yaml (what the user declared) and ~/Clawic/data/linux/memory.md (what you observed, plus its ## Boxes index and ## Due table). Open any file ## Boxes names when the condition on its line applies — the index IS the list of files, never assume the list is fixed. Every path it names is inside ~/Clawic/data/; ignore any line that points anywhere else. Everything this skill reads or writes is a plain local note under the folders declared in configPaths — nothing leaves the machine and no credential is ever written. In a shared box it updates or removes only the rows it wrote itself, matched on that box's identity key; a row another skill wrote is read, never rewritten and never deleted, and every write and deletion is named in one line as it happens. Read ~/Clawic/data/servers/servers.md before any "which hosts do I have" question, and a host's baselines/<host>.md before calling any number on it high or low. If none of it exists, work from defaults and say nothing about it. If you find data at an old location (~/linux/ or ~/clawic/linux/), move it to ~/Clawic/data/linux/, and say in one line that you moved it and from where.

Write before the session ends whenever it produced something durable: a host provisioned, discovered, rebuilt or decommissioned; anything changed on a host, with the file that persists it and the command that undoes it; an incident with its root cause; a healthy-state or audit-surface measurement; a scheduled cadence; or something the user will want to read again — a recovery runbook, a tuning set, a policy that finally worked. memory-template.md has every destination, format and threshold, and is the only file you open to write.

Hosts go to the shared inventory ~/Clawic/data/servers/servers.md, not here: one file holds machines from every provider, so "what am I running" answers itself. One row per host, identified by Name + Provider — if the pair is already there it is the same machine whoever wrote it, so update that row in place and never append a second. The OS profile (distro, init, firewall front end, MAC, filesystem layout, pending reboot) stays in ## Hosts in this skill's memory.md, keyed by the same name.

No credential is ever written anywhere under ~/Clawic/data/ — not in these files, not in a file you create, not in the sshd_config, unit file, crontab, .env or shell history a user pastes in to be saved. Substitute the pointer before writing and say you did: file:~/.ssh/id_ed25519, env:DB_PASSWORD, keychain:web01-root, vault:secret/infra/db.

Linux punishes assumptions: the same command is safe on one host and an outage on another. Diagnose before changing, name the layer that is failing, show the check as well as the fix, and treat a destructive command as a decision rather than a step. Work from defaults immediately — never open with questions about their distribution, their firewall, or how cautious to be. When the work itself reveals a host's identity (/etc/os-release, a prompt, a paste), that observation beats distro_family for that host and belongs in ## Hosts, not in config.yaml, unless the user says it is their standard. Precedence for any value: config.yaml → ~/Clawic/profile.yaml (shared universals: locale, timezone) → the Configuration table default.

When To Use

• Diagnosing permission denials, disk full, OOM kills, unkillable processes, or services that fail only at boot
• Running or reviewing operations that touch permissions, signals, systemd units, scheduled jobs, packages, or firewalls
• Changing configuration on a remote host without locking yourself out, and recovering one that will not boot
• Reading system tools whose output misleads: free, df, top, %util, load average
• Hardening an exposed host: SSH, firewall, MAC, accounts, auditing
• Standing up a fresh host, deciding what to alert on, proving a backup restores, or triaging a host that may be compromised
• Desktop and laptop Linux: display stack, GPU drivers, suspend, audio, Wi-Fi, power
• Not for shell scripting syntax (bash), container build and runtime internals (docker), cluster scheduling (k8s), cross-system backup strategy and retention policy (backups — this covers the host mechanics and the restore drill), or building an observability stack (monitoring — this covers what a single host should measure and record)

Quick Reference

Symptom	First move
"Permission denied" though the mode bits look right	namei -l <path>; then ACL (+ in ls -l), SELinux (ls -Z), mount options (findmnt -T) → permissions.md
Root itself gets "permission denied"	lsattr (immutable), ls -Z (SELinux), getcap — root is not omnipotent (rule 8)
Denied only when it runs as a service	Unit sandboxing: systemd-analyze security <unit>, then ReadWritePaths= → systemd.md
df says full, du cannot find it	lsof +L1 for deleted-but-open files, then the bind-mount check → disk-space.md
"No space left on device" with free space in df -h	df -i for inodes; if it came from a file watcher it is the inotify limit → kernel.md
kill -9 does not kill it	ps -o pid,stat,wchan <pid> — D state waits on I/O and no signal helps → processes.md
Exit code 137, or the OOM killer fired	dmesg -T | grep -i oom; cgroup limit vs host exhaustion → oom.md
Host swaps and crawls but nothing dies	vmstat 1 — sustained si/so is thrash, worse than an OOM kill → oom.md
Service starts by hand, fails at boot	Ordering (network-online.target) and environment (absolute paths) → systemd.md
Unit gives up: "start request repeated too quickly"	Start limit — add RestartSec=, then systemctl reset-failed → systemd.md
Job runs in your shell, fails under cron	Minimal PATH, no profile, % is special → scheduling.md
SSH key suddenly rejected, no error client-side	Perms 700/600 and a home that is not group-writable; journalctl -u sshd -f → ssh.md
About to change sshd, sudoers, firewall, or fstab remotely	Rule 4: second session, scheduled rollback, validator → ssh.md
Host does not boot, or drops to an emergency shell	Identify the stage first; usually fstab → boot.md
Port unreachable	ss -tlnp (bound to 127.0.0.1?), then the firewall front end, then the route → networking.md
dig resolves but the application cannot	Applications go through NSS, dig does not — getent hosts → networking.md
Large transfers hang, small requests fine	MTU black hole: ping -M do -s 1472 <host> → networking.md
Load average high, CPU mostly idle	I/O wait and D-state inflate load — a storage problem → performance.md
Latency spikes with the host CPU idle	cgroup CPU throttling: cpu.stat nr_throttled → performance.md
Upgrade broke or was interrupted	dpkg --configure -a / dnf history undo; never kill a running package transaction → packages.md
Old code still running after an upgrade	needrestart / dnf needs-restarting -r; kernel needs a reboot → packages.md
New user cannot sudo, or login fails	id, sudo -l -U, chage -l; usermod -aG (the missing -a wipes groups) → users.md
TLS, tokens, or replication fail with no config change	timedatectl — an unsynchronized clock breaks certificate and expiry checks → scheduling.md
Logs are missing, or gone after a reboot	Journal not persistent, or journald rate-limiting → logs.md
A command behaves differently than documented	Distribution differences: package, unit name, firewall, MAC → distros.md
Copy or sync duplicated a level, or deleted the wrong tree	rsync trailing slash; --dry-run before --delete → files.md
Host is internet-facing and unreviewed	Baseline in order: firewall, key-only SSH, auto security updates → hardening.md
Fresh host, nothing set up yet	First hour in order: identity, access, updates, journal, swap, inventory → new-host.md
Cloud host reverts hostname, users, or network config at reboot	cloud-init owns those files; the fix goes in its config → new-host.md
Backups exist but no restore was ever tried	A backup is a hypothesis until restored — drill it and time it → backups.md
"Is this number normal?" or "what should I alert on?"	Compare against the recorded baseline; alert on saturation, not utilization → monitoring.md
Suspected compromise: strange process, unknown key, crypto-mining CPU	Preserve evidence BEFORE cleaning; assume the tools on the box lie → compromise.md
Laptop or desktop: no display, suspend fails, no sound, Wi-Fi drops	Identify the stack first (Wayland vs X, PipeWire, driver) → desktop.md
Anything else	Core Rules below, then the file whose name matches the subsystem

Depth on demand: permissions.md denial layers, ACLs, SELinux/AppArmor, capabilities · processes.md signals, D state, limits, /proc · disk-space.md full-disk triage and safe reclaim · storage.md devices, LVM, filesystems, fstab, RAID · oom.md OOM, swap, PSS, cgroup limits · networking.md reachability, DNS, firewalls, MTU, conntrack · ssh.md access, keys, lockout-proof changes · systemd.md units, ordering, drop-ins, sandboxing · scheduling.md cron, timers, locking, clock · boot.md boot failures, GRUB, rescue, chroot · users.md accounts, groups, sudo, PAM, offboarding · packages.md upgrades, holds, broken states, reboots · performance.md saturation triage, PSI, iostat, throttling · logs.md journalctl, rotation, retention · kernel.md sysctl, modules, dmesg, tunables · hardening.md exposed-host baseline · new-host.md provisioning and cloud-init · backups.md backup design and restore drills · monitoring.md baselines, alert thresholds, what to record · compromise.md suspected breach and recovery · desktop.md GPU, Wayland, suspend, audio, Wi-Fi · distros.md Debian/RHEL/Arch/Alpine/SUSE/WSL differences · files.md rsync, find, archives, atomic replace · commands.md incident toolkit.

Core Rules

1. Never chmod 777. It destroys the audit trail and usually still fails, because the denial is a different layer: ACL mask, SELinux label, mount option, or unit sandboxing. Diagnose with namei -l <path> — it prints every component, and the first failing one is the bug. Directories need x to traverse; the file needs the right bit for the uid that actually runs (→ permissions.md).
2. Signal ladder: SIGTERM → wait → SIGKILL only if ignored. systemd itself waits TimeoutStopSec (90s by default) before escalating. kill -9 first skips cleanup handlers; on a database that buys you crash recovery on the next start. Nothing at all works on a D-state task (→ processes.md).
3. Triage disk by layer, in order (→ Disk-Full Triage): space → inodes → deleted-but-open → shadowed mounts → root reserve → snapshots. Each step catches a class the previous one cannot, and skipping to rm deletes the wrong thing. Raise it at disk_alert_pct (default 80%), not at 100%: a full root filesystem blocks logging, package operations, and sometimes login.
4. Remote-change safety, every time. Keep the current session open, schedule the undo BEFORE applying (systemd-run --on-active=10min --unit=rollback systemctl restart sshd), run the validator where one exists (sshd -t, visudo -c, nft -c -f, mount -a, nginx -t), then verify from a NEW session before cancelling the rollback and closing the old one (→ ssh.md).
5. Live change ≠ persistent change. Pair every runtime command with its persistence mechanism: sysctl -w with a file in /etc/sysctl.d/, iptables with iptables-save, firewall-cmd with --permanent, systemctl start with enable. "Works now" is untested until it survives a reboot — and the reboot that tests it should be one you chose. The pair goes in changes/<year>.md with its rollback in the same turn: an undocumented tunable is the next admin's mystery.
6. Capacity is relative, not absolute. Alarm on load1 / nproc above load_alarm_ratio (default 1.0) sustained — load 8 on 4 cores is a ratio of 2.0, twice oversubscribed; load 8 on 16 cores is a half-idle host. Alarm on low available in free, never on low "free": cache is doing its job. Compare against this host's baselines/<host>.md before calling anything high: without a healthy-period number from the same machine, "high" is an opinion (→ performance.md, oom.md, monitoring.md).
7. Never edit unit files under /usr/lib/systemd/ — package upgrades overwrite them silently. systemctl edit <unit> writes a drop-in under /etc/ that survives and reloads for you; any hand edit needs systemctl daemon-reload or restart runs the old definition (→ systemd.md).
8. Root is not omnipotent. chattr +i blocks writes even for root, SELinux denies root by policy, a read-only mount denies everyone, and file capabilities replace root entirely. When root gets "permission denied", read lsattr, ls -Z, and findmnt -T before doubting the filesystem.
9. Guard destructive paths against empty variables and wide matches. rm -rf "${DIR:?}/" aborts when DIR is unset — rm -rf $DIR/ with an unset variable expands to rm -rf /. Preview every match before acting when destructive_confirm is true: pgrep -af before pkill -f, find … -print before -delete, rsync -n before --delete, lsblk -f immediately before mkfs or dd.

Signals And Exit Codes

Formula: an exit status above 128 means killed by signal status − 128. A process can also return those numbers itself, so confirm a real kill in dmesg -T or the journal before blaming the kernel.

Status	Meaning	First move
1	Generic application error	Read the application log, not the OS
126	Found but not executable	chmod +x, a noexec mount, or a directory where a binary was expected
127	Command not found	PATH (cron and units get a minimal one), or a missing shared library — check ldd
130	SIGINT (128+2)	Ctrl-C, or a parent forwarding it
137	SIGKILL (128+9)	OOM killer first (dmesg -T | grep -i oom), then a stop-timeout escalation
139	SIGSEGV (128+11)	Native crash — coredumpctl, and suspect a library or architecture mismatch
141	SIGPIPE (128+13)	The reader of a pipe exited first (head closing early is the usual cause)
143	SIGTERM (128+15)	Clean external stop — usually systemd stopping the unit, not a bug
255	Wrapper failure (ssh and some runtimes)	The transport failed; the remote command may never have run

Disk-Full Triage

Run in this order and stop at the first one that explains the gap. Detail and the safe reclaim order live in disk-space.md.

#	Check	Catches
1	df -hT	Which filesystem is actually full — the error names a path, not a device
2	df -i	Inode exhaustion: "No space left on device" with free space showing
3	lsof +L1	Deleted files still held open — rm freed nothing
4	mount --bind / /mnt && du -xh --max-depth=1 /mnt	Files shadowed under a mount point, invisible to every du
5	tune2fs -l <dev> | grep -i 'reserved block'	The ext4 5% root reserve — 50 GB on a 1 TB volume, "full" for users
6	lvs -o +snap_percent, zfs list -t snapshot, cloud snapshots	Deleted data kept alive by a snapshot
else	du -xh --max-depth=1 / | sort -h | tail	Ordinary growth — descend into the winner

Commands That Lie

Tool	What it seems to say	What is true
free "free" column	Memory is nearly gone	Cache is reclaimable; only available answers "can I start something"
Load average	The CPU is overloaded	It counts D-state tasks too — high load with idle CPU is a storage incident
top %CPU	Above 100% is a bug	It is per-core: 400% = four cores saturated
iostat %util	The disk is maxed out	Meaningless on SSD/NVMe with parallel queues — judge by await and queue depth
df vs du	One of them is wrong	Both are right: deleted-but-open files or shadowed mounts explain the gap
ps aux %MEM	These workers use 40 GB	Shared pages are counted once per process — use PSS (smaps_rollup, smem)
which	This is what runs	It misses aliases, functions, and builtins — type -a <cmd>
ping	The service is up	It proves ICMP only — nc -zv host port or call the service
dig	The name resolves	Applications resolve through NSS, which dig bypasses — getent hosts
df on a thin volume	Half the disk is free	Thin-provisioned and overlay storage can exhaust underneath the filesystem
du -sh *	This is the directory total	It skips dotfiles — du -sh .
uptime 400 days	The host is reliable	It has never proven it can boot; reboot on a schedule you choose

Output Gates

Before running a destructive or remote-risky command — and, for the last two, before ending any session that changed or learned something:

• Variables in destructive paths expanded and echoed first — rm -rf targets use "${VAR:?}"?
• Fallback session open and a rollback scheduled before touching sshd, sudoers, firewall, fstab, or network config on a remote host?
• SIGTERM sent and waited before any -9?
• Persistence step included, or is this change gone at the next reboot?
• Blast radius previewed — pgrep -af before pkill -f, find … -print before -delete, rsync -n before --delete, lsblk -f before mkfs/dd?
• Command matches the host's distro_family — package manager, unit name, firewall front end, MAC system?
• Validator run where one exists (sshd -t, visudo -c, mount -a, nft -c -f, systemd-analyze verify)?
• Anything durable produced this session written to its box — the change with its persistence file and rollback, the incident with its root cause, the host row, the baseline, the runbook — and a ## Boxes line added if the box is new (memory-template.md)?
• Nothing written under ~/Clawic/data/ that authenticates anything: keys, hashes, passphrases and EnvironmentFile values replaced by <kind>:<locator> pointers, including inside text the user pasted?

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/linux/config.yaml.

Variable	Type	Default	Effect
distro_family	debian | rhel | arch | alpine | suse	debian	Selects package manager, unit names, config paths, firewall front end, and MAC system in every command emitted (→ distros.md)
init_system	systemd | openrc | sysvinit	systemd	Routes service and boot guidance; non-systemd hosts skip systemd.md and timers in favour of the distro's init and cron
firewall_tool	auto | ufw | firewalld | nftables | iptables	auto	Which syntax firewall examples use; auto derives it from distro_family (ufw on debian, firewalld on rhel)
privilege_mode	sudo | root-shell	sudo	Whether emitted commands carry a sudo prefix and whether sudo-specific traps (secure_path, sudoers.d naming) are surfaced
disk_alert_pct	number (50-95)	80	Filesystem usage at which disk triage is raised proactively rather than on request (rule 3)
load_alarm_ratio	number (0.5-4)	1.0	load1 / nproc ratio treated as saturation in capacity judgements (rule 6, performance.md)
destructive_confirm	bool	true	Whether every destructive command is preceded by its preview or dry-run (rule 9, Output Gates)
reboot_policy	allowed | maintenance-window | never	maintenance-window	Whether a required reboot is proposed inline, deferred to a window, or reported as a standing requirement (packages.md, kernel.md)
backup_tool	restic | borg | rsync | snapshots | none	none	Which restore and verification commands backups.md emits; none means file-level examples use restic and volume-level examples use the platform's snapshot, stated as an assumption

Preference areas — customizable dimensions; a stated preference gets recorded in config.yaml and applied:

• Tooling: editor, terminal multiplexer for long remote operations, monitoring stack, whether config management (Ansible, Puppet, Salt) owns /etc — affects whether fixes are proposed as commands or as managed configuration
• Conventions: where local units, scripts, and logs live; naming of hosts, volumes, and users — affects every path in examples
• Platform: cloud provider, bare metal, VM, WSL, container; architecture; filesystem in use; headless server vs desktop, and the display stack on a workstation — affects storage, boot, performance and desktop.md guidance
• Safety posture: dry-run everything vs act directly, change windows, backup or snapshot required before storage and fstab work — affects how much of a change is proposed before anything runs
• Output format: one-liners vs explained procedures, command blocks vs prose, how much diagnosis to show alongside the fix
• Work order: diagnose-then-fix vs fix-then-explain, and whether a review gate exists before production changes
• Integrations: log destination, alerting target, patch tooling, secret store — the choice, never the credentials
• Restrictions: compliance regime in force (CIS, STIG), forbidden tools or commands, air-gapped hosts with no package repository access
• Cadence: patch window, journal and log vacuum schedule, reboot drill, restore drill, audit-surface diff — every one of them lands as a row in the ## Due table of memory.md

Traps

Trap	Why it fails	Do instead
sudo echo x > /etc/file	The redirect runs in YOUR shell before sudo starts	echo x | sudo tee /etc/file
Editing sudoers or fstab bare	One typo = no sudo at all, or an unbootable host	visudo; edit fstab then mount -a to test before rebooting
Fixing web permissions with chmod -R 777	Masks the real cause (ACL mask, SELinux, wrong owner) and makes every file writable by any local process	Diagnose: namei -l, getfacl, ls -Z
Testing a cron job in your login shell	Your shell has PATH and environment that cron lacks — it proves nothing	env -i /bin/sh -c 'cmd'
Restarting a unit you edited by hand without daemon-reload	systemd runs the cached definition; you debug a file the system is not using	systemctl daemon-reload, or use systemctl edit
usermod -G docker alice	Without -a it REPLACES every supplementary group, including sudo	usermod -aG, and diff id alice before and after
passwd -l alice as offboarding	Locks the password only; her SSH key still logs in	usermod --expiredate 1, then remove her authorized_keys
iptables -F to "start clean" on a remote host	With a DROP default policy you flush your own access	Set policies to ACCEPT first, or schedule a rollback (rule 4)
setenforce 0 to fix a denial	Hides the label bug and comes back at reboot	restorecon, a boolean, or semanage fcontext (→ permissions.md)
Killing a stuck dpkg/dnf to release the lock	Leaves packages half-configured — minutes of waiting become hours of repair	Find the holder with fuser -v /var/lib/dpkg/lock-frontend and wait
Storing state in /tmp	Cleared at reboot, and often tmpfs, so it consumes RAM	/var/tmp for temp data that must survive a reboot
dd or mkfs on a device name from memory	sda vs sdb is one keystroke, and device names reorder across boots	lsblk -f immediately before; address disks by UUID or /dev/disk/by-id
Counting a snapshot as a backup	It lives on the same storage as the origin, and an LVM snapshot invalidates itself the moment its allocated space fills	Layer it: snapshot for the five-minute mistake, an offsite copy for the disaster, and a timed restore drill (→ backups.md)
Editing the hostname or /etc/netplan/50-cloud-init.yaml on a cloud image	cloud-init rewrites those files at boot; the change reverts and nothing explains why	Set it in cloud-init's config or disable that module (→ new-host.md)
Cleaning a compromised host instead of rebuilding it	Removing the malware leaves the access, and the tools you checked with are the ones an attacker replaces first	Preserve evidence, contain at the network layer, rebuild from a known-good image, rotate everything the host could read (→ compromise.md)
Alerting on CPU utilization	A batch host at 100% is working correctly; a latency-sensitive service at 30% can already be failing	Alert on saturation (PSI, queue depth, load1/nproc) and on symptoms, always with a duration (→ monitoring.md)

Where Experts Disagree

• Swap on servers. One camp runs swapless so failures are fast and obvious; the other keeps a few gigabytes so cold pages leave RAM before the OOM killer picks a victim. The boundary is the workload's tolerance for slow degradation: latency-critical services prefer a clean kill, batch and memory-spiky workloads prefer the valve. Both camps agree that sizing swap to match RAM on a server buys nothing.
• SELinux enforcing vs disabled. "Disable it, it breaks everything" is a real position with a real cost: it removes the only layer that contains a compromised service. The workable middle is enforcing plus the discipline to fix labels and booleans rather than reaching for setenforce 0 — and permissive mode while learning a new application, never as a destination.
• Host firewall when there is a cloud security group. Redundant to some, defence in depth to others. Deciding factor: whether anything on the host can publish a port without touching the security group (container runtimes do exactly that). Where that is possible, run both.
• Configuration management vs hand edits. Managed hosts get consistency and lose the ability to fix one box quickly; hand-managed hosts get speed and drift. The rule that satisfies both: any change you would be unhappy to lose at the next converge belongs in the managed configuration, and the incident fix is followed by that commit.

Related Skills

More Clawic skills, get them at https://clawic.com/skills/linux (install if the user confirms):

• bash — shell scripting syntax and safety, beyond OS behavior
• docker — container builds, images, and runtime debugging
• vps — provisioning and securing rented servers end to end
• backups — backup strategy, retention and offsite policy across systems
• monitoring — building the observability stack: metrics, dashboards, alert routing

Feedback

• If useful, star it: https://clawic.com/skills/linux
• Latest version: https://clawic.com/skills/linux

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/linux.