Detect Attacks Early: Monitoring File Changes, Logins, and Resource Anomalies

The later a security incident is handled, the higher the cost. Discovering a trojanized site after a week versus discovering it on the same day results in vastly different remediation difficulties—the former assumes the attacker has already done significant damage, while the latter only requires closing a single entry point.

The following monitoring measures are low-cost but can compress the “time to detection” from weeks to the same day.

1. Critical File Change Monitoring

PHP files in the website directory should not change under normal circumstances (except during updates). Any unexpected modification is a strong signal of compromise.

Method 1: Periodic Comparison (Simplest)

# Generate baseline (do this when the system is clean)
find /var/www/html -type f -name '*.php' -exec md5sum {} ; | sort -k2 > /root/baseline.md5

# Periodic comparison
find /var/www/html -type f -name '*.php' -exec md5sum {} ; | sort -k2 > /tmp/now.md5
diff /root/baseline.md5 /tmp/now.md5 && echo "No changes detected"

# Simpler approach: Only check recently modified files
find /var/www/html -type f -name '*.php' -mtime -1 -print
# If there is output, verify if you made the changes yourself

“Recently modified PHP files” is the most practical metric; it requires no baseline setup and can be run immediately.

Method 2: File Integrity Tools

# AIDE (Linux File Integrity Checking Tool)
yum install -y aide
aide --init
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# Regular checks
aide --check

# Add to cron job
0 4 * * * /usr/sbin/aide --check | mail -s "File Integrity Check" ops@example.com

2. Login Monitoring

# ① View recent logins
last | head -20
lastb | head -20          # Failed login attempts

# ② Count failed logins (signal of brute force)
lastb | awk '{print $3}' | sort | uniq -c | sort -rn | head

# ③ View currently online users
w
who

# ④ Alert on login (or periodic summary)
# Simple method: Run last once via crontab and filter unknown IPs
0 * * * * last -n 20 | grep -v 'KnownIP' | mail -s "Login Records" ops@example.com

Key Focus: SSH Key Files

# Writing an unknown public key to authorized_keys = backdoor access
# This file must be included in change monitoring
md5sum ~/.ssh/authorized_keys
ls -l ~/.ssh/authorized_keys

# Regularly check content
cat ~/.ssh/authorized_keys

3. Cron Job and Startup Item Monitoring

Backdoors need persistence, almost always involving cron or systemd. Therefore, changes here are the most valuable signals.

# Generate baseline
crontab -l > /root/cron.baseline
systemctl list-unit-files --state=enabled > /root/services.baseline

# Periodic comparison
diff <(crontab -l) /root/cron.baseline && echo "No cron changes"
diff <(systemctl list-unit-files --state=enabled) /root/services.baseline && echo "No service changes"

# Lighter weight: Regularly check for newly added services
find /etc/systemd/system -name '*.service' -mtime -7

“New systemd services” are the most easily overlooked persistence mechanism. Many people only check crontab and miss this area.

4. Resource Anomalies (CPU Spikes)

The most typical sign of crypto-mining malware is long-term high CPU usage.

#!/bin/bash
# /root/check-load.sh
LOAD=$(awk '{print $1}' /proc/loadavg)
CORES=$(nproc)
THRESH=$(echo "$CORES * 2" | bc)

if (( $(echo "$LOAD > $THRESH" | bc -l) )); then
  echo "Load anomaly: $LOAD (Cores: $CORES)"
  ps aux --sort=-%cpu | head -10
fi

*/10 * * * * /root/check-load.sh | mail -s "Load Alert" ops@example.com

5. Outbound Connection Monitoring

A normal web server should not have many strange outbound connections. Abnormal outbound traffic is a strong signal of compromise.

# View established outbound connections
ss -antp state established | awk '{print $5}' | cut -d: -f1 
  | sort | uniq -c | sort -rn | head -20

# Focus on connections to uncommon ports/unknown IPs
ss -antp state established | grep -vE ':80|:443|:22|:3306'

6. Anomalies in Web Logs

# Find scanning behavior (same IP generating many 404s)
awk '$9 == 404 {print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# Find sensitive path probing
grep -iE '.env|.git|phpmyadmin|wp-login|xmlrpc' /var/log/nginx/access.log 
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

See the article “Finding Abnormal Access from Access Logs” for details. Here we emphasize: check regularly, not just after an incident occurs.

Monitoring Summary

Decision Tree

Establish Security Monitoring (by Cost-Effectiveness)
├─ ① Recently modified PHP files (Cheapest, strongest signal)
├─ ② Cron and Systemd service changes (Persistence requires this)
├─ ③ Authorized_keys content
├─ ④ CPU anomalies (Typical sign of mining)
├─ ⑤ Abnormal outbound connections (Strong sign of compromise)
└─ ⑥ Login records and failed attempts

After Detecting Anomalies
├─ First confirm if it was your own change
└─ If not → Treat as intrusion event (processes/mechanisms/accounts/passwords)

Scenario Simulation: Intrusion Occurred Three Weeks Ago, Detected Today

Phenomenon: An abnormal file was found during a routine check.
Investigation revealed the earliest modification occurred three weeks ago—during which time all monitoring showed no alerts.

1. Determine why it wasn’t detected initially →
   The monitoring items were “site accessibility” and “resource usage,”
   both of which remained completely normal during the intrusion.
2. Identify the ignored signal category →
   File changes, abnormal logins, and unexpected scheduled tasks were never monitored.
3. Add file integrity detection →
   Establish baselines for directories that shouldn’t change and alert on modifications. This is the most direct measure.
4. Add login and privilege escalation detection →
   Monitor for logins from unusual sources or new accounts with login capabilities.
5. Incorporate stealthier states into monitoring →
   Scheduled task lists and dynamic library preload files should trigger immediate alerts if non-standard content appears.
6. Verify end-to-end functionality with a drill →
   Create a controlled change to verify the alert chain works fully.

# ① Baseline + Integrity (for directories that shouldn't change)
find /website_dir -name '*.php' -newermt 'yesterday' -printf '%TY-%Tm-%Td %TH:%TM  %pn' | sort

# ② Login and Account Anomalies
last -10
awk -F: '$3>=1000 && $7 !~ /nologin|false/ {print $1,$3,$7}' /etc/passwd

# ③ Stealthy points: This file should normally be empty
cat /etc/ld.so.preload 2>/dev/null && echo '!! Content exists, verify source'

# ④ Check for unexpected entries in cron jobs
crontab -l -u root; ls -la /etc/cron.d/

Root Cause: Monitoring only covered “availability” and “resources.”
The intrusion left no trace in these metrics. By the time it was discovered, three weeks had passed.

Two Key Takeaways:

• Availability monitoring cannot detect intrusions: A compromised site can remain fully accessible.
  You must separately monitor “file changes” and “abnormal logins.”
• After configuring detection, validate end-to-end availability with a controlled drill.
  The most reliable method is to actively create a change and verify the alert arrives.

8 Common Pitfalls

1. Only checking after an incident: At that point, the scope is uncontrollable.
2. Checking only crontab, not systemd: Persistence mechanisms vary.
3. Not monitoring authorized_keys: Attackers can return anytime.
4. Creating baselines after compromise: Baselines must be generated in a clean state.
5. Alert fatigue causing ignored warnings: Set reasonable thresholds.
6. Ignoring outbound connections: A strong signal of compromise.
7. Failing to update baselines after legitimate updates: Leads to continuous false positives.
8. Not investigating the entry point upon anomaly detection: Cleaning without patching leads to recurrence.

Real Customer Feedback (From Cloud Market Reviews and Inquiries for Similar Products)

The following excerpts are taken from real buyer reviews and inquiries for similar operations services on Alibaba Cloud / Tencent Cloud / Huawei Cloud Cloud Markets (providers include our store and competitors, provided solely as reference for real feedback, not fictional tickets from this site):

1****8 Rating: 5.0 Points 2019-10-29 22:35 Good 1

1****1 Rating: 5.0 Points 2020-06-18 11:50 1****1 Rating: 5.0 Points 2020-04-26 18:46 1****1 Rating: 5.0 Points 2020-04-26 18:45 1****1 Rating: 5.0 Points 2020-04-26 18:45 1****1 Rating: 5.0 Points 2020-04-26 18:45 1****1 Rating: 5.0 Points 2020-04-26 18:44 He****She Rating: 5.0 Points 2019-09-26 09:47