The biggest waste in incident handling is guessing. Restarting, reinstalling, or rebooting without information often fails to fix the issue and destroys critical evidence. This page outlines our troubleshooting sequence and the immediate steps you can take when an incident occurs.
This section covers the emergency response part of Server Management. For website-level issues, please refer to Website Maintenance.
Three Immediate Actions
- Preserve the Scene
Do not restart immediately. Take screenshots, record error messages, and save logs (dmesg, nginx/apache error logs, journalctl). Restarting clears a significant amount of critical diagnostic information.
- Confirm the Scope
Is it affecting a single machine, one service, one user, or everything? Have there been recent changes (upgrades, configuration changes, IP changes, software installations)? Most incidents are related to recent changes.
- Restore First or Investigate First?
If business continuity is critical, restore first, but at least preserve some scene information. Many “resolved” issues are actually just cases where the evidence was wiped out.
Common Incidents and Diagnosis Paths
| Symptom | Possible Causes (Sorted by Probability) | Key Checkpoints |
|---|---|---|
| Website inaccessible, 502 Bad Gateway | Backend service down / PHP-FPM processes exhausted / Database connection failure | Service status, error logs, connection counts, database reachability |
| Website 500 Internal Server Error | PHP fatal errors / Out of memory / File permission issues | PHP error log, memory_limit setting, directory ownership |
| Suddenly slow but low load | Disk IO saturation / Memory exhaustion causing swap usage / Slow database queries | iostat, free + vmstat, slow query logs |
| Disk full | Log rotation not configured / Uncleaned temporary files / Unremoved backups / Accumulated database binlogs | Use du to locate large directories, check if inodes are exhausted |
| Memory exhaustion | Process limit exceeded / Service memory leak / No swap protection configured | Top RSS-consuming processes, services with frequent restarts |
| Frequent reboots / RAID degraded | Unstable power supply / CPU overheating / Hardware failure | IPMI logs, SMART data, RAID controller events, temperature and fan status |
| Hard drive alert | Abnormal SMART status / Growing bad sectors / Vibration issues | Trends in key SMART attributes (do not rely solely on pass/fail), vendor tool diagnostics |
| No network connectivity | NIC failure / Switch port issue / Upstream line problem / Routing misconfiguration | Layered ping tests, NIC status, confirmation from the data center side |
| Under attack / Abnormal traffic | DDoS / Scanning / Vulnerability exploitation | Traffic trends and source distribution, WAF logs, connection counts |
| Certificate expired | Auto-renewal not configured / Renewal failed / Path changed | Actual certificate validity period, renewal task logs, last execution result |
General Principles for Troubleshooting
Check the Timeline First
If the incident start time aligns with the most recent change, it is highly likely caused by that change. System log timestamps may be inaccurate, so verify the timezone first.
Layered Localization
Physical layer (hardware, links) → System layer (resources, processes) → Service layer (applications, logs). Skipping directly to application logs wastes time at the wrong level.
Change Only One Variable at a Time
If you change three things simultaneously and the problem disappears, you won’t know which change worked, leading to repeat mistakes. Change one thing, verify once.
Investigate Root Cause After Fixing
“It works after reboot” does not mean it is resolved. You must explain why it failed, why there was no alert, and how to prevent recurrence. This will be included in the monthly report.
Preparation You Can Do Now
- Ensure you have an accessible management channel (IPMI / iDRAC / iLO, or cloud console VNC) — SSH may be unreachable during an incident.
- Maintain a recoverable backup and know how to restore it.
- Keep an archive of key roles and port lists (who manages what, contact info, open services).
- Ensure monitoring alerts are actually monitored, not sent to unread email inboxes.
Frequently Asked Questions
How quickly can you respond to emergencies?
For clients with 7×24 response agreements, we handle urgent incidents (business interruption, data loss risk, security events) immediately. Specific response times are subject to the contract; pre-signing, we clearly define what constitutes an emergency.
Do you handle hardware replacements?
We are responsible for diagnosis, coordination, and verification. The hardware itself is handled by the manufacturer or data center. We assist in locating slots, coordinating on-site or remote replacement, and perform rebuilding and verification post-replacement. Negotiation processes and responsibility divisions are clarified before signing.
Can I just get ad-hoc incident support without a long-term contract?
Yes, billed per incident. However, please note: if the environment has inherent configuration defects (no monitoring, unusable backups, lack of documentation), fixing it once will likely lead to recurrence. We will explicitly warn you of this risk.