Troubleshooting 100% CPU/Memory Usage: Command Checklist from Process Identification to Mining Detection

When a server becomes nearly unresponsive, many people’s first reaction is to “restart it.” While restarting does restore functionality, it simultaneously wipes the scene—you’ll never know what consumed all the resources, so the issue will likely recur.

Troubleshooting isn’t complicated; the key is to first check the overall status, then pinpoint the process, and finally determine if that process should exist. Below are the commands arranged in this order.

Step 1: Check Overall Status

# Load averages (1/5/15 minutes)
uptime
cat /proc/loadavg

# CPU and Memory Overview
top
# Or for a more intuitive view:
htop

# Memory Details
free -m

Understanding free -m is crucial, as many people misinterpret it here:

              total        used        free      shared  buff/cache   available
Mem:           3935        1200         180          40        2555        2400

Although used looks high, buff/cache is memory used by the kernel for caching, which is automatically released when needed. The column you should actually look at is the last one, available—this represents “how much is still usable”. Many cases of “full memory” are actually misreadings.

Step 2: Pinpoint Processes

# Sort by CPU usage
ps aux --sort=-%cpu | head -15

# Sort by Memory usage
ps aux --sort=-%mem | head -15

# Show only key columns
ps -eo pid,ppid,user,%cpu,%mem,etime,comm --sort=-%cpu | head -15

# Resource details for a specific process
top -p PID

# Files opened by the process / connections made (to identify abnormal processes)
lsof -p PID
ls -l /proc/PID/exe
ls -l /proc/PID/cwd

Step 3: Determine if the Process Should Exist

Typical Characteristics of Mining Processes

• CPU stays near 100% for long periods, and is unrelated to business traffic (full even at midnight).
• Process names are random or disguised as system processes (one extra or missing character).
• Executable files are hidden in /tmp, /dev/shm, or hidden directories.
• Sustained outbound connections to unknown IP ports.
• Automatically reappears after being killed (indicating a resurrection mechanism).

# Check outbound connections (miners always connect to pools)
netstat -antp | grep ESTABLISHED
ss -antp | grep ESTAB

# Find executables in hidden and temporary directories
ls -la /tmp /dev/shm /var/tmp
find / -name '.*' -type f -perm -u+x 2>/dev/null | head

# Check resurrection mechanisms (root cause of auto-restart after kill)
crontab -l
ls -l /etc/cron.d/
systemctl list-unit-files --state=enabled

High Resource Usage on the Database Side

# Currently running queries
mysql -e "SHOW PROCESSLIST;" | head -30

# Whether connection limits are reached
mysql -e "SHOW STATUS LIKE 'Threads_connected'; SHOW VARIABLES LIKE 'max_connections';"

# Slow queries
mysqldumpslow -s t -t 10 /var/log/mysql/slow.log

OOM (Out of Memory) Consequences

# Check if the kernel has killed any processes
dmesg | grep -i -E 'oom|killed process'
journalctl -k | grep -i oom

If you see Out of memory: Kill process ... (php-fpm), it means memory is insufficient. At this point, do not just increase the process count—if process count × single-process memory exceeds available memory, OOM will trigger even faster.

Decision Tree

Resources Fully Used
├─ High CPU
│  ├─ php-fpm / nginx usage → Business concurrency, check slow queries and process count
│  ├─ mysqld usage          → Slow queries, indexes, connection count
│  ├─ Unknown process + outbound connections → Mining: Block outbound → Check resurrection mechanism → Clean files → Change passwords
│  └─ Auto-restarts after kill → Must have cron / service / startup item, clean it first
├─ High Memory
│  ├─ free available is sufficient → Just cache usage, not an issue
│  ├─ Too many php-fpm processes    → Recalculate pm.max_children based on memory
│  └─ dmesg shows OOM               → Memory is truly insufficient, add RAM or reduce process count
└─ High Load but Low CPU
   └─ Likely I/O wait → Check disk with iostat / iotop

Scenario Walkthrough: CPU Full, Process Restarts Minutes After Killing

Symptoms: A server’s CPU is consistently near full capacity, causing significantly slower business response times.
After identifying and terminating the highest resource-consuming process, usage drops immediately but returns to full capacity within minutes.

1. Check load first, not just CPU → Confirm it is process consumption rather than I/O wait,
   This step rules out disk bottlenecks masquerading as CPU issues.
2. Record the full command line and path of the process → The path points to a temporary directory,
   and the process name deliberately mimics a system process. Write this down before taking action, otherwise clues disappear when the process vanishes.
3. Understand why it “reappears” → If ending the process doesn’t make it disappear permanently,
   it indicates an automatic resurrection mechanism; find it instead of repeatedly killing the process.
4. Check scheduled tasks → A cron job exists that downloads and executes an external script.
5. Remove the resurrection mechanism first, then kill the process → Order matters: cut off the source first,
   and the process will not reappear.
6. Continue investigating other entry points → Check accounts, keys, and startup items,
   Only after confirming no second resurrection point exists is the handling considered complete.

# First check load and CPU composition (%wa high indicates I/O wait, not true CPU busy)
top -bn1 | head -5
iostat -x 1 3 2>/dev/null | tail -5

# Record the highest resource-consuming process, along with its full command line and path
ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -15
ls -l /proc/PID/exe 2>/dev/null
cat /proc/PID/cmdline | tr '' ' '; echo

# Find resurrection mechanism (do this step before killing the process)
crontab -l -u root; ls -la /etc/cron.d/
systemctl list-unit-files --type=service | grep enabled

Root Cause: This is not a normal performance issue, but a program planted to continuously consume resources,
kept alive by scheduled tasks. Repeatedly killing the process is ineffective because the resurrection mechanism remains active.

Two Key Takeaways:

• If a process restarts after being killed, check the resurrection mechanism; do not kill it a second time.
  The order should be “cut off the source, then clean the process,” otherwise you are just repeating the same action.
• Before taking action, record the process’s command line, executable file path, and parent process.
  Once the process ends, this information disappears, yet it is precisely the basis for determining its nature (normal service vs. malicious program).

8 Common Pitfalls

1. Simply restarting to resolve issues: Wipes the scene, guaranteeing recurrence.
2. Misreading used in free: Look at available; buff/cache is automatically reclaimed.
3. Only killing mining processes: Without cleaning cron jobs, they auto-resurrect in seconds.
4. Blindly increasing php-fpm process count: Triggers OOM faster.
5. Ignoring outbound connection checks: netstat is the most direct way to identify mining.
6. Cleaning up without changing passwords: Entry points remain, leading to quick recurrence.
7. Assuming high load is always a CPU issue: It could also be I/O wait; check the wa value in top.
8. Failing to preserve the scene: Save a snapshot of ps aux and network connections before troubleshooting; post-mortem analysis relies on them.

Real Customer Feedback (From Cloud Market Reviews and Inquiries for Similar Products)

The following excerpts are taken from real buyer reviews and inquiries for similar operations services on Alibaba Cloud / Tencent Cloud / Huawei Cloud Cloud Markets (service providers include our store and peers, provided solely for reference of real feedback, not fictional tickets from this site):

1****4 Rating: 5.0 Score 2023-05-08 11:53 1****3 Rating: 5.0 Score 2022-09-22 22:01 r****u Rating: 5.0 Score 2020-07-07 15:05

b****4 Rating: 5.0 Score 2025-06-23 16:57 b****4 Rating: 5.0 Score 2025-06-23 16:56 Jian****Wei Rating: 5.0 Score 2021-01-20 15:03 Technical skills are awesome! Speed is very fast! It turned out to be a trojan causing Baidu global hijacking!!!!1 f****u Rating: 5.0 Score 2021-01-02 20:43 Thanks for assisting in maintaining Yuncai’s normal operation. 1****1 Rating: 5.0 Score 2020-11-20 10:03 Consistently good service as always 1****1 Rating: 5.0 Score 2020-11-17 09:13 Thanks for the consistently good service. 1****1 Rating: 5.0 Score 2020-09-25 15:42 Many thanks. 1****1 Rating: 5.0 Score 2020-08-24 15:34 Many thanks to your company’s technical staff 1****1 Rating: 5.0 Score 2020-07-24 09:49 Many thanks! 1

d****9 Rating: 5.0 Score 2023-02-15 09:11 Yong****Ying Rating: 5.0 Score 2021-03-29 11:04