Linux on-disk forensics #

Logs

Linux logging is messy because there are two parallel systems: traditional text logs under /var/log/ (syslog, auth.log, messages - appended by rsyslog/syslog-ng) and the newer systemd journal (binary files under /var/log/journal/, queried with journalctl). Many distros run both, but Debian 12+, Fedora and Arch are journal-only by default - check whether a syslog daemon is installed before looking for the text files. Timestamp semantics differ: text logs record only wall-clock (and often local time without a TZ suffix, the single biggest Linux timeline trap); the journal carries both a wall-clock and a monotonic timestamp, letting you re-order events even if someone turned the system clock back.

User / shell artifacts

Filesystem timestamp reality check: Modern Linux defaults to relatime mount option - atime only updates if (a) atime < mtime/ctime, or (b) atime is >24h older than now. On noatime volumes (common for busy servers, flash) atime never updates. Always check cat /proc/mounts at collection; otherwise your "last access" findings are wrong. ext4 and XFSv5 give you crtime (birth) via stat - ext2/ext3 do not. Attackers using touch -r reference target clone the mtime+atime of reference onto target; the ctime still updates because the inode changed, which is how you catch it.

AI agent artifacts

AI coding agents and assistants run commands, read files and edit code as the user, so their transcripts are often the best execution log on the host - the prompt, every command, and the output the agent saw. The same files collect any secrets the agent read. Paths below are for Linux; ~ = /home/<user> (and /root).

What to look for in the transcripts:

• Commands it ran - timeline every shell tool call next to EDR process events.
• Secrets it read - .env, cloud credential files, SSH keys, kube configs, browser profiles. What was read is in the transcript, so the transcript is now a copy of the secret.
• Where data went - web fetches, curl, git push to unfamiliar remotes, new MCP servers pointing at external URLs.
• Safety switches off - --dangerously-skip-permissions, --yolo, --trust-all-tools, broad allow rules, auto-approve. Check the settings file's mtime against the incident window.
• Instructions the user didn't write - prompt injection arrives in tool output (a web page, issue, README or MCP response). Read the tool results just before an odd action.
• Persistence - a planted CLAUDE.md, AGENTS.md, GEMINI.md, .cursor/rules, hook or MCP server re-runs the attacker's instructions every session.
• Agents driven by malware - the August 2025 Nx "s1ngularity" npm compromise ran claude, gemini and q with permission checks off to hunt for secrets. The parent is a package script, not a terminal, and there's no matching user prompt. Record agent and extension versions too: a poisoned Amazon Q VS Code release (1.84.0, July 2025) shipped a wipe prompt.

Inventory (then copy; take SQLite files with their -wal / -shm files, or close the app first - recent rows often live only in the WAL. Check that your KAPE / Velociraptor / UAC targets actually cover these folders):

for h in /home/* /root; do
  for t in .claude .claude.json .codex .gemini .copilot .continue .cursor .codeium .kiro .aws/amazonq .ollama .lmstudio \
           .config/Code/User .config/Cursor/User .config/Windsurf/User; do
    [ -e "$h/$t" ] && ls -ld "$h/$t"
  done
done 2>/dev/null
ls -ld /etc/claude-code /usr/share/ollama/.ollama /tmp/qlog 2>/dev/null
find / -xdev -name .aider.chat.history.md 2>/dev/null

Persistence

Linux persistence lives in three big buckets: scheduled execution (cron, systemd timers, at), process-start hooks (shell rc files, PAM, LD_PRELOAD, dynamic linker tricks), and kernel-level (loadable modules, eBPF programs). Below is the ASEP catalog you should sweep on every engagement. "Where to find it" matters more than "what tool to run" - attackers often customize to evade the obvious tools.

Scheduled execution

• cron - The classic Unix scheduler. System-wide tabs live at /etc/crontab and in the /etc/cron.{hourly,daily,weekly,monthly} drop-dirs; files added to /etc/cron.d/ are included. Per-user tabs live at /var/spool/cron/crontabs/<user> (Debian/Ubuntu) or /var/spool/cron/<user> (RHEL/Fedora). Format is 5-field schedule + user (system crontab only) + command. Attackers love it because any user can schedule in their own crontab without privs, and the entry just looks like a one-liner. Trap: crontab -l shows only the invoking user's tab - you must iterate every user. anacron (/etc/anacrontab) handles missed jobs on laptops; also check it.
• at / batch jobs - One-shot scheduled commands via at; queued in /var/spool/cron/atjobs/. Rarely used operationally, so a single at job on a server is highly suspicious. List with atq.
• systemd units (services) - The modern Linux init system. A "unit" is a text file (.service, .timer, .socket, .path, .mount, etc.) that systemd parses and acts on. Service units specify what to run (ExecStart=), under which user (User=), and when (WantedBy=multi-user.target = run on boot). Three directory tiers (in load order): /etc/systemd/system/ (admin), /run/systemd/system/ (runtime), /usr/lib/systemd/system/ (distro). Per-user units in ~/.config/systemd/user/ run only when that user is logged in / lingering. Attackers drop a .service file named like a legit daemon, enable it (systemctl enable creates a symlink in the target wants-dir), and reboot survives. Hunt: systemctl list-unit-files --state=enabled, diff against a known-good distro baseline; systemd-analyze security <unit> for sandboxing weaknesses.
• systemd timers - The modern cron replacement. A .timer unit says "every N minutes" or "at specific times" and activates a matching .service. More flexible than cron (random delays, monotonic schedules, calendar expressions) and easier to miss. systemctl list-timers --all is mandatory.
• systemd generators - Executables in /etc/systemd/system-generators/, /usr/lib/systemd/system-generators/, or /run/systemd/system-generators/ run very early during boot, before most units, and can dynamically generate unit files. Niche attacker persistence that fires before many defenses are up.
• systemd .path units - Trigger a service when a filesystem path changes. Attacker-friendly: "run my payload every time /tmp/.X-flag is touched."
• systemd user lingering - loginctl enable-linger <user> keeps a user's systemd instance running even when they're not logged in. Combined with a per-user service, that's reboot-surviving code execution as a non-privileged user.

Legacy init (still present on older/RHEL systems)

• /etc/rc.local - A plain shell script run late in boot (if the rc-local.service stub is enabled). Lives on through modern systemd distros for backwards compatibility.
• /etc/init.d/* + /etc/rc?.d/ - SysV init scripts and their symlinked runlevel directories. Mostly superseded, but still invoked on systemd distros via the systemd-sysv-generator. New files in /etc/init.d/ are a red flag on any system from the last 10 years.
• /etc/inittab - The classic SysV init config. Legacy; still present on older distros / embedded.

Shell & login hooks (user-level, no privs)

• Shell init files - ~/.bashrc, ~/.bash_profile, ~/.profile, ~/.zshrc, ~/.zprofile, /etc/profile, /etc/bash.bashrc, /etc/profile.d/*.sh. Any of these runs when a user starts an interactive shell (bashrc = non-login, bash_profile/profile = login). Attackers append a line that executes a payload, downloads a C2 stub, or exports an LD_PRELOAD. If a user SSHs in once a day, that's your schedule.
• MOTD / update-motd.d - Scripts in /etc/update-motd.d/ execute at every interactive login (SSH / console) to generate the message of the day. Writable by root; a new script here runs as root every login. Audit /etc/motd too (static) and /etc/pam.d/sshd for the pam_motd module.
• XDG autostart - Desktop-environment persistence: ~/.config/autostart/*.desktop and /etc/xdg/autostart/*.desktop. Only triggers when a GUI session starts, so applicable to workstations more than servers.
• Default login shell swap - Changing a user's shell in /etc/passwd to a wrapper that runs the payload and then execs bash.

Dynamic linker & library-level hooks

• /etc/ld.so.preload - Any shared object listed in this plain-text file is loaded into every dynamically-linked ELF process on the system before any other library. Classic rootkit hook (Jynx, Azazel, Symbiote all use it). The malicious .so hooks libc functions (readdir, open, stat) to hide its own files, processes, and connections. A non-empty /etc/ld.so.preload on a server is almost always malicious.
• LD_PRELOAD environment variable - Same mechanism, scoped to a process and its children. Attackers set it in ~/.bashrc or /etc/environment so it applies to every interactive user session. SUID binaries ignore it (good) but anything else honors it.
• LD_LIBRARY_PATH injection - Prepending an attacker-controlled directory to LD_LIBRARY_PATH lets malicious libs shadow real ones. Set via /etc/environment, user shell rc, or systemd unit's Environment=.
• Shared library trojaning - Replace or patch a system-wide library (libc.so.6, libpam.so, libssl.so) with a backdoored version. Detect by signature check against distro packages: debsums -c (Debian/Ubuntu), rpm -Va (RHEL/Fedora), AIDE, Tripwire.

Kernel-level

• Loadable Kernel Modules (LKMs) - .ko files in /lib/modules/$(uname -r)/; loaded via modprobe / insmod. Diamorphine, Reptile, LKM-Rootkit hook syscalls to hide processes, files, and network connections. Enumerate with lsmod, cat /proc/modules, inspect /proc/kallsyms for suspicious symbols. dmesg | grep -i taint - any taint flag means a non-distro module got loaded. /etc/modules-load.d/ and /etc/modprobe.d/ are config dirs that cause modules to auto-load at boot.
• eBPF-based rootkits - BPFDoor, Pumakit, Boopkit, TripleCross, Symbiote (hybrid). Modern alternative to LKMs because eBPF programs are loadable at runtime by root, don't need a kernel-matching compile, and bypass many LKM-focused detections. Hunt via bpftool prog list, bpftool map list, bpftool link list; inspect /sys/fs/bpf/ for pinned programs; cat /sys/kernel/debug/tracing/kprobe_events for attachment points. Tracee and Falco can detect anomalous eBPF loads at runtime.
• Kernel core_pattern - /proc/sys/kernel/core_pattern defines what to do with core dumps; setting it to a pipe (|/tmp/evil.sh %p) means your script runs as root every time any process crashes. Classic container-escape primitive too.

Authentication & access-granting primitives

• SSH authorized_keys - The simplest, most durable, most-often-missed attacker persistence on Linux. Any public key appended to ~/.ssh/authorized_keys grants that user shell access. Attackers also drop into /root/.ssh/authorized_keys (if they have root) and sometimes /home/*/\.ssh/authorized_keys across every user. Sweep every home directory. Check for ssh keys that reference an "options=" prefix (command=, no-pty) hiding what they do.
• sshd_config backdoors - /etc/ssh/sshd_config and the newer /etc/ssh/sshd_config.d/*. Attacker-interesting directives: AuthorizedKeysFile pointed at a world-readable path they control; ForceCommand that runs their shim before dropping to a shell; Match User blocks that relax rules for specific users; PermitRootLogin yes; PubkeyAuthentication yes + a planted key; Banner set to a payload path (rare but seen). Also check ~/.ssh/config, ~/.ssh/authorized_keys options like command=, environment=.
• SSH host-key or client-config swap - Replacing /etc/ssh/ssh_host_* to accept attacker clients, or planting a ~/.ssh/config that proxies through an attacker host.
• PAM (Pluggable Authentication Modules) - /etc/pam.d/* are text files describing the auth stack per service (sshd, login, sudo, passwd, …). Each line names a .so module. Attackers either (a) add a line like auth sufficient pam_backdoor.so that always returns success, or (b) replace a legit module (pam_unix.so) with a backdoored version that logs cleartext credentials to disk (classic Skidmap, SkidMAP, variants of Tsunami). pam-rhino, Pambd are reference malicious modules. Enumerate loaded module paths with ldd on each listed .so; checksum them against the distro package.
• sudoers manipulation - /etc/sudoers and /etc/sudoers.d/*. An entry like attacker ALL=(ALL) NOPASSWD: ALL grants pw-less root. More subtle: Defaults !tty_tickets, Defaults env_keep += "LD_PRELOAD", allowing specific binaries that have GTFO privesc paths. Always check for sudoers includes with recent mtime.
• SUID / SGID binaries - Files with the setuid or setgid bit run with the owner's privileges regardless of invoker. find / -perm -4000 -type f 2>/dev/null. Cross-reference every unusual SUID against GTFOBins - any match is a privesc gadget. Attackers also drop their own SUID-root shell copy somewhere obscure (/var/tmp/.cache/sh) as a durable local privesc path.
• File capabilities - A finer-grained alternative to full SUID; capability bits attached via setcap. getcap -r / 2>/dev/null. Dangerous ones to see on unexpected binaries: cap_sys_admin, cap_sys_ptrace, cap_dac_override, cap_net_raw, cap_setuid. Example: python3 with cap_setuid+ep = instant root.

Package manager & config hooks

• apt hooks - /etc/apt/apt.conf.d/*. Lines like DPkg::Pre-Install-Pkgs or APT::Update::Post-Invoke execute commands as root around every package install/update. Attacker-perfect because sysadmins update frequently.
• dnf/yum plugins - /usr/lib/python*/site-packages/dnf-plugins/*.py; equivalent code-execution opportunity on RHEL-family.
• Debian dpkg triggers / RPM scriptlets - Package maintainer-scripts (pre-install, post-install, pre-remove, post-remove) run as root. Backdoored packages are a supply-chain concern (XZ Utils CVE-2024-3094).
• Udev rules - /etc/udev/rules.d/ can trigger scripts on hardware events. Example: a USB insert that execs a payload. Rare but seen in physical-access attacks.
• NetworkManager / dispatcher scripts - /etc/NetworkManager/dispatcher.d/* run on network state changes (as root).
• logrotate hooks - /etc/logrotate.d/* allows prerotate/postrotate shell blocks; runs as root daily.

Rootkits & fileless

Modern Linux malware increasingly runs without touching disk - either as an eBPF program loaded into the kernel's virtual-machine subsystem (visible to bpftool but invisible to ls), or as a process backed by an in-memory file descriptor created with memfd_create, where /proc/<pid>/exe resolves to /memfd:... or shows (deleted). The classic LKM rootkits (Diamorphine, Reptile) are still around but now compete with eBPF-based alternatives (BPFDoor, Pumakit) that are harder to detect because eBPF is a legitimate observability feature of the kernel. Hunting here is a game of comparing enumerations: process from /proc walk vs. ps, loaded modules vs. /proc/modules vs. sysfs, and auditing which eBPF programs are loaded against a known-clean baseline.

• eBPF-era rootkits: BPFDoor, Symbiote (LD_PRELOAD), Pumakit, boopkit, TripleCross. Hunt via bpftool prog list, cat /sys/kernel/debug/tracing/kprobe_events, unexpected tracepoints.
• Fileless execution - memfd_create: look for /proc/<pid>/exe → /memfd:…, fexecve, process whose /proc/<pid>/exe is (deleted) but still mapped (/proc/<pid>/maps).
• io_uring - bypasses many hooks; watch for unusual io_uring_setup in auditd.
• Hidden processes - compare /proc enumeration to ps; LKM rootkits (Diamorphine) hide via syscall hook.

Containers

Containers are a namespace + cgroup construct, not a hypervisor boundary - they share the host kernel, so a container compromise can escalate to host compromise under the right conditions (privileged flag, host-pid, mounted docker.sock, kernel LPE). Forensics splits into three layers: the image (what shipped - scan with Trivy/Grype, verify provenance), the runtime (what actually executed - Falco/Tracee eBPF events, container stdout/stderr, docker events), and the orchestration plane (K8s audit log, admission events, ServiceAccount token usage). The containerd/overlay2 directory tree persists the writable layer of every container that ran, so you can often reconstruct a destroyed container's filesystem from /var/lib/docker/overlay2/ even after docker rm.

• Docker - /var/lib/docker/ (containers, image, overlay2, network). docker inspect, docker events --since, daemon.json.
• Containerd - /var/lib/containerd/, task shim sockets.
• K8s - kubectl get events --all-namespaces, audit logs on API server, admission controller trail, serviceaccount token mount (/var/run/secrets/kubernetes.io/serviceaccount/token).
• Escapes to know by CVE: runC CVE-2024-21626 (Leaky Vessels), cgroups v2 release_agent abuse (older), docker.sock mount, privileged + --pid=host, CAP_SYS_ADMIN + user namespace tricks.
• Image provenance - skopeo inspect, SBOM, signature (cosign) when evaluating a suspect image.

Linux CVEs to check exposure against

• CVE-2022-0847 Dirty Pipe - pipe splice r/w on read-only files
• CVE-2023-0386 OverlayFS - fuse privesc
• CVE-2023-4911 Looney Tunables - glibc GLIBC_TUNABLES local root
• CVE-2024-1086 nf_tables UAF - kernel LPE
• CVE-2024-3094 XZ Utils backdoor (Jia Tan) - sshd supply chain; versions 5.6.0/5.6.1

Linux collection tooling

• UAC - Unix-like Artifacts Collector (Thiago Canozzo Lahr). Script-based triage, modular, outputs tar. The "KAPE of Linux"; Velociraptor's Linux artifacts and offline collector cover the same ground and scale across a fleet.
• AVML (Microsoft) - kernel-agnostic memory capture
• LiME - kernel-module memory capture (requires matching kernel-headers)
• Volatility3 Linux profiles (via dwarf2json / ISF)
• osquery live triage; Velociraptor Linux artifacts; GRR
• Loki/Thor YARA scanning; Sysmon for Linux (ETW-like)