Troubleshooting Compromised Servers (Zombie Machines): Processes, Cron Jobs, Startup Items, and Accounts

When a server is compromised to act as a “zombie machine” (botnet node), the most common symptom is consistently high CPU usage, followed by abnormal outbound connections. The difficulty in handling this issue lies in the fact that: killing the process alone is ineffective because it will respawn—there is another mechanism launching it.

Therefore, troubleshooting must cover four key areas: Processes, Cron Jobs, Startup Items, and User Accounts.

① Processes

# Sort by CPU usage to identify resource hogs
top
ps aux --sort=-%cpu | head -15

# View full command lines (paths and arguments are critical)
ps auxww | head -20

# Check the executable path of a process
ls -l /proc/PID/exe
cat /proc/PID/cmdline | tr '' ' '; echo

# Check external connections opened by the process
ss -antp | grep PID
lsof -p PID

# Find abnormal established outbound connections
ss -antp state established | awk '{print $5, $7}' | sort | uniq -c | sort -rn | head -20

# Check for listeners on non-standard ports
ss -lntp

Hidden Processes

# Compare PIDs from ps with those in /proc (hidden processes appear in /proc but not in ps)
ls /proc | grep -E '^[0-9]+$' | sort -n > /tmp/proc_pids.txt
ps -e -o pid= | sort -n > /tmp/ps_pids.txt
comm -23 /tmp/proc_pids.txt /tmp/ps_pids.txt

If PIDs are output, hidden processes exist (usually due to replaced system commands or loaded malicious modules). In such cases, a full OS reinstallation is recommended.

② Cron Jobs (Scheduled Tasks)

This is the most common reason for processes respawning after being killed.

# Crontab for current user and root
crontab -l
crontab -u root -l
ls -l /var/spool/cron/ /var/spool/cron/crontabs/

# System-level cron
cat /etc/crontab
ls -l /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.weekly/

# Review all cron files
find /etc/cron* /var/spool/cron -type f -exec echo '=== {}' ; -exec cat {} ;

# systemd timers
systemctl list-timers --all

Typical characteristics: A scheduled task that downloads and executes scripts from external sources, or periodically writes to specific files.

# Find tasks involving download and execution
grep -rE 'curl|wget|bash -c|/tmp/' /etc/cron* /var/spool/cron 2>/dev/null

③ Startup Items

# systemd services
systemctl list-unit-files --state=enabled
ls -l /etc/systemd/system/ | head -30

# Traditional rc.local
cat /etc/rc.local
ls -l /etc/init.d/

# Profile scripts (triggered upon login)
cat /etc/profile
ls -l /etc/profile.d/

# Find recently added service units
find /etc/systemd/system /usr/lib/systemd/system -name '*.service' -mtime -30

④ Accounts and Keys

# List users (look for unfamiliar accounts)
cat /etc/passwd | awk -F: '$3 >= 1000 {print $1, $3}'

# Check for newly added sudo privileges
grep -rE 'ALL=(ALL' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

# Login keys (Critical! Attackers often inject their own public keys)
cat ~/.ssh/authorized_keys
ls -l /root/.ssh/ /home/*/.ssh/

# Recent login records
last | head -20
lastb | head -20      # Failed login attempts

authorized_keys is the most critical file to check. After gaining access, attackers often write their own public keys here, allowing them to log in even if passwords are changed.

⑤ Dynamic Library Hijacking

# Check for LD_PRELOAD hijacking
cat /etc/ld.so.preload 2>/dev/null
echo $LD_PRELOAD

# If /etc/ld.so.preload contains content, it points to libraries preloaded into all processes

This type of hijacking infects all commands, including the troubleshooting tools themselves (ps, top may fail to show abnormal processes). In such cases, a full OS reinstallation is recommended.

Remediation Order

① Disconnect network or restrict outbound traffic (prevent further external connections and spawning)
② Clean up cron jobs and startup items    ← Do this first, otherwise processes will respawn
③ Terminate abnormal processes
④ Delete malicious files
⑤ Clean up authorized_keys and unauthorized accounts
⑥ Change all credentials
⑦ Patch the entry point used for intrusion
⑦ Re-check the next day

Order matters: Cut off the “spawning mechanism” before killing processes. If done in reverse, processes will restart immediately, making it seem like they cannot be killed.

# Emergency: Temporarily block outbound traffic (example, adjust based on environment)
iptables -A OUTPUT -p tcp --dport 3333 -j DROP

# Stop and disable malicious systemd services
systemctl stop ServiceName
systemctl disable ServiceName
rm -f /etc/systemd/system/ServiceName.service
systemctl daemon-reload

Decision Tree

Abnormal High CPU / Abnormal Outbound Connections
├─ Use top to find high-resource processes
│  ├─ Found and path is suspicious → Continue below
│  └─ Not found (hidden) → Proceed with OS reinstallation
├─ Check cron jobs (crontab / cron.d / systemd timer)
│  └─ If abnormal → Clean up first, then kill processes
├─ Check startup items (systemd / rc.local / profile.d)
├─ Check accounts and authorized_keys
├─ Check /etc/ld.so.preload (for hijacking)
└─ Remediation Order: Block outbound → Remove spawn mechanisms → Kill processes → Delete files
   → Clean keys/accounts → Change credentials → Patch entry point → Re-check next day

Scenario Walkthrough: CPU at 100% Long-term, Process Respawns After Killing

Symptom: A server’s CPU remains at full capacity for an extended period, causing extremely slow business response.
Identifying the highest resource-consuming process and terminating it restores normal performance temporarily, but the CPU hits 100% again within minutes.

1. Analyze load composition first to rule out I/O wait masquerading as CPU issues → Wait percentage is low, confirming active computation.
2. Gather evidence before taking action → Record command lines, executable locations, and parent processes,
   as this information disappears once the process ends.
3. Analyze process characteristics → Executables located in temporary directories and names mimicking system processes indicate malware, not legitimate services.
4. Understand why it respawns → Repeatedly killing the process is ineffective, indicating a persistence mechanism;
   find this mechanism instead of continuing to kill the process.
5. Trace the source via cron jobs → A scheduled task was found downloading and executing code periodically.
6. Break the mechanism first, clean the process second, then secure channels → Correct order prevents recurrence.

# Distinguish between true CPU load and I/O wait
top -bn1 | head -5          # Look at %wa
iostat -x 1 3 2>/dev/null | tail -5

# Gather evidence: Command line and executable path
ps -eo pid,ppid,user,%cpu,etime,cmd --sort=-%cpu | head -10
ls -l /proc/SuspiciousPID/exe 2>/dev/null
tr '' ' ' < /proc/SuspiciousPID/cmdline; echo

# Find spawning mechanisms (must be done before killing processes)
crontab -l -u root; ls -la /etc/cron.d/
systemctl list-unit-files --type=service | grep enabled
ls -lt /etc/systemd/system/ | head

Root Cause: A resource-intensive program is automatically spawned and kept alive by a cron job. Killing the process repeatedly is futile because the spawning mechanism persists.

Two Key Takeaways:

• If a process keeps respawning, do not kill it again; investigate the spawning mechanism directly.
  The correct order is “cut off the source, then clean the process,” otherwise you are just repeating the same action.
• Gather evidence before handling abnormal processes.
  Command lines and executable paths are crucial for determining intent; this data is lost once the process terminates.

Locations and Commands for Five Types of Hiding Spots

An easily overlooked aspect is the order: Check cron jobs before killing processes.
Many mining processes respawn minutes after being killed because a cron job is still configured to launch them.
Cutting off the spawning mechanism first, then cleaning the process, ensures a permanent fix.

# Cron jobs (check every user individually, do not just check root)
crontab -l -u root
ls -la /etc/cron.d/ /etc/cron.daily/ /var/spool/cron/
cut -d: -f1 /etc/passwd | xargs -I{} crontab -l -u {} 2>/dev/null

# Recently added services in startup items
systemctl list-unit-files --type=service | grep enabled
ls -lt /etc/systemd/system/ | head -20

# Dynamic library hijacking (any output indicates a problem)
cat /etc/ld.so.preload 2>/dev/null && echo "!! ld.so.preload exists, verify source"

# Login-capable accounts and recent login records
awk -F: '$3>=1000 && $7 !~ /nologin|false/ {print $1, $3, $7}' /etc/passwd
last -20

The /etc/ld.so.preload item deserves special attention: Libraries specified here are preloaded into all processes.
It is commonly used to hide processes and files (a common rootkit technique).
Normally, this file should not exist. If it contains content, treat it as an intrusion signal even if you don’t understand the contents.

8 Common Pitfalls

1. Killing processes without cleaning mechanisms: They restart immediately.
2. Incorrect remediation order: Spawning mechanisms should be cut off before killing processes.
3. Neglecting authorized_keys: Attackers can still log in.
4. Ignoring systemd timers: Checking only crontab leads to missed threats.
5. Using infected commands for troubleshooting: Results are unreliable (due to LD_PRELOAD hijacking).
6. Failing to change passwords: The entry point remains open.
7. Not restricting outbound traffic: External connections and downloads continue.
8. Skip re-checking: Cannot confirm if the system is fully cleaned.