The most common problem during an incident is having no process, only reactions—whoever sees it first acts, multiple people change configurations simultaneously, and in the end, nothing is fixed and no one knows what was changed.
The value of a simple but clear process lies in: ensuring that “restoring service” and “finding the root cause” happen sequentially, rather than mixed together.
Core Principle: Stop the Bleeding First, Then Find the Root Cause
This is the most important rule in the entire process.
The primary goal of incident handling is to restore service, not to find the cause.
The problem with fixing and investigating simultaneously is: you might preserve the fault scene to “figure it out,” extending downtime; or repeatedly try fixes without understanding the issue, where each attempt may introduce new variables.
Correct order:
- First, use a definitive method to restore service (rollback, restart, switch back to the old environment)
- After service is restored, calmly investigate the root cause
Six Stages
| Stage | Action | Output |
|---|---|---|
| ① Detection | Alert triggered or user feedback | Confirm “there is really a problem” |
| ② Grading | Impact scope and severity | Determine if immediate全员 response is needed |
| ③ Mitigation | Prioritize service restoration | Service available |
| ④ Localization | Find root cause | Determine the cause |
| ⑤ Fix | Fundamental resolution | No recurrence |
| ⑥ Review | Record and improve | Timeline + Improvement items |
① Detection: Confirm It’s Real
# Multi-channel confirmation to avoid single-point false alarms
curl -I https://example.com/ -m 10
# Test again from another machine
# Check monitoring history (is it really down, or just probe jitter?)Confirm there is really a problem before acting. Probe jitter and transient network interruptions can trigger alerts; blind responses waste effort.
② Grading: How Big Is the Impact?
| Level | Criteria | Response |
|---|---|---|
| Critical | Entire site unavailable / Risk of data loss | Handle immediately, notify relevant parties |
| General | Partial function abnormality | Handle during working hours |
| Mild | Does not affect main workflow | Schedule for handling |
# Quickly judge impact scope
curl -I https://example.com/ # Homepage
curl -I https://example.com/key-page # Core functions
ss -lntp # Are all services running?
systemctl --failed # Any failed services?③ Mitigation: Restore First
| Scenario | Mitigation Method |
|---|---|
| Recent changes made | Rollback changes (fastest and most certain) |
| Service unresponsive | Restart the service |
| Issues during migration/switchover | Switch back to old environment |
| Single node failure | Remove the node |
| Under attack | Rate limit / Enable protection |
# Rollback example (WordPress plugin causing white screen)
wp plugin deactivate plugin-name
# Rollback database (confirm backup exists)
wp db import backup-date.sql
# Switch back to old environment (change DNS resolution, note TTL)
# Or temporarily modify hosts for verification“Rolling back the most recent change” is the mitigation method with the highest success rate. Incidents are highly correlated with changes in time; suspect it first.
④ Localization: Find the Root Cause
# After service is restored, investigate calmly
journalctl -p err --since '2 hours ago' --no-pager
tail -100 /var/log/nginx/error.log
tail -100 /var/log/php-fpm/error.log
# Compare with change records: What was changed recently?
# This is often faster than reading logs line by line“What was changed recently” is the most efficient clue. It is much faster than reading logs line by line, so change records must be maintained regularly.
⑤ Fix and ⑥ Review
What to Record in the Review
Incident Review Record Template
## Timeline (Critical!)
- HH:MM Alert triggered / User feedback
- HH:MM Fault confirmed, graded
- HH:MM Mitigation actions (specifically what was done)
- HH:MM Service restored
- HH:MM Root cause identified
- HH:MM Fundamental fix completed
## Impact
- Duration, impact scope (which functions/how many users)
## Root Cause
- Direct cause + Why this path was taken
## What Went Right
- Which actions shortened recovery time
## Improvement Items (Most Important)
- Specific action + Owner + Deadline
Example: Add monitoring alert for XX (Zhang San, this week)
Example: Supplement rollback documentation for XX (Li Si, next week)The timeline is the core of the review. Without a timeline, you cannot know “which action actually worked” nor measure recovery efficiency.
Decision Tree
Incident Occurs
├─ ① Confirm it's real (multi-channel verification, exclude probe jitter)
├─ ② Grade (Entire site down = Critical, handle immediately)
├─ ③ Mitigation ← Prioritized over finding root cause
│ ├─ Recent changes? → Rollback changes (highest success rate)
│ ├─ Migration/Switchover? → Switch back to old environment
│ └─ Service unresponsive? → Restart the service
├─ ④ Investigate root cause after service is restored
│ └─ First check "what was changed recently", then check logs
└─ ⑤ Review
├─ Timeline (Must record)
└─ Improvement items must have owners and deadlines (otherwise they won't be implemented)Scenario Simulation: Three Hours of Incident Handling, No One Can Explain What Happened During Review
Phenomenon: An online incident took three hours to recover.
When trying to conduct a post-incident review later, it was found that: there were no complete time records, no one could clearly explain what operations were performed at the time,
and it was impossible to determine when the initial alert was triggered.
- First, admit that what is missing is not conclusions but records → Without timestamps,
it is impossible to judge which segment dragged on the longest, and improvement items cannot be proposed. - Reconstruct the timeline from logs → Try to restore as much as possible from monitoring, chat records, and command history.
Due to missing data, only a rough version can be obtained. - Identify sequence issues in the handling process → Mitigation actions were too late;
a large amount of time was spent earlier on finding the root cause. - Clarify principles for Level 1 incidents → When business is completely unavailable,
the goal is restoration, not查明原因 (finding the cause); methods like restart, rollback, and traffic switching can be used first. - Write “preserve evidence before acting” into the process →
Mitigation actions destroy the scene; solidify the state at that time before taking action. - Require records to include real timestamps →
And archive them after resolution in four parts: timeline, impact scope, root cause, and improvement items.
# Solidify the scene into a file before acting—mitigation destroys evidence
{
echo "Time: $(date '+%F %T')"
echo "--- Executed ---"
uptime; free -m; df -h | tail -3
ss -lntp | head -10
echo "--- Log Snippet ---"
journalctl --since '30 min ago' -p err | tail -30
} | tee /tmp/handover-$(date '+%F-%H%M').txt
# Use system time for the timeline, do not fill in later from memory
date '+%F %T'Root Cause: Lack of a process for “preserving evidence first, deciding whether to mitigate based on level, and keeping full timestamps.”
Without these, reviews rely on memory, and improvement items become impossible to discuss.
Two points worth remembering:
- For Level 1 incidents, restore first, then find the cause; restoration actions do not need to solve the root cause.
Restart, rollback, and traffic switching are legitimate means, provided the scene is preserved first. - Review records must include real timestamps.
The sequence and duration filled in later will almost certainly be distorted, and these numbers are exactly the basis for judging which segment needs improvement most.
Incident Grading and Response Correspondence
| Level | Criteria | Response Requirement | Mitigation Before Root Cause |
|---|---|---|---|
| Level 1 | Core business completely unavailable | Immediate response, restore first | Yes |
| Level 2 | Main functions affected, workarounds exist | Respond ASAP | Yes |
| Level 3 | Local function abnormality, main body available | Handle as planned | Depends on situation |
| Level 4 | Experience issues, no business loss | Schedule for handling | No |
The purpose of grading is not classification, but determining whether to skip the root cause first.
The only goal of a Level 1 incident is to restore business; methods like restart, rollback, and traffic switching (which do not solve the root cause but can restore service) can be used;
Levels 3 and 4 are not worth using these methods because the cost may outweigh the benefit.
Four Items to Record in the Review
- Timeline: When discovered, when responded to, when restored.
Record real timestamps; memories filled in later are usually inaccurate. - Impact Scope: Which functions, how many users, how long.
- Root Cause: Write to “why it happened,” not stop at “what broke.”
- Improvement Items: Actionable steps + Owner + Deadline.
Improvement items without an owner and deadline are equivalent to not writing them.
# Preserve evidence during incident: Save the scene first, then act
uptime; free -m; df -h
ss -tnp | head -30
tail -200 /var/log/nginx/error.log
journalctl --since '1 hour ago' -p err | tail -100
# Use system time for the timeline, do not rely on memory
date '+%F %T'The rule “preserve evidence before acting” must be written into the process because mitigation operations destroy the scene.
Restart, rollback, and traffic switching are fine, but save a copy of the load, connections, and log snippets at that time first;
otherwise, post-incident reviews can only rely on speculation, the root cause item can only be written as “suspected,” and improvement items become difficult to address.
8 Common Pitfalls
- Fixing and investigating simultaneously: Extends downtime.
- Multiple people changing configurations simultaneously: Unclear what was changed.
- Not rolling back recent changes: Skipping the method with the highest success rate.
- Not recording the timeline: Unable to review afterwards.
- No grading: Treating small matters as big, big matters as small.
- Not reviewing after fixing: The same incident will happen again.
- Improvement items without owners: Will never be implemented.
- Acting without confirming alerts: Wasting effort on probe jitter.
Real Customer Feedback (From Cloud Market Similar Product Reviews and Consultations)
The following excerpts are from real buyer reviews and consultations for similar O&M services on Alibaba Cloud / Tencent Cloud / Huawei Cloud Cloud Markets (service providers include our store and peers, for reference only, not fictional tickets on this site):
1****1 Rating: 5.0 Points 2020-06-18 11:51 Many thanks! He****She Rating: 5.0 Points 2020-04-21 15:04 Boss is great! Configuration was fast! No pauses in between! Good, good!
b****4 Rating: 5.0 Points 2025-06-23 16:57 b****4 Rating: 5.0 Points 2025-06-23 16:56 Jian****Wei Rating: 5.0 Points 2021-01-20 15:03 Technology is awesome, speed is fast! It turned out to be a Trojan causing global Baidu hijacking!!!!1 f****u Rating: 5.0 Points 2021-01-02 20:43 Thanks for assisting in maintaining normal Yuncai operations. 1****1 Rating: 5.0 Points 2020-11-20 10:03 Consistently good service 1****1 Rating: 5.0 Points 2020-11-17 09:13 Thanks for the consistently good service. 1****1 Rating: 5.0 Points 2020-09-25 15:42 Many thanks. 1****1 Rating: 5.0 Points 2020-08-24 15:34 Many thanks to your company’s technical staff 1****1 Rating: 5.0 Points 2020-07-24 09:49 Many thanks! 1
b****4 Rating: 5.0 Points 2024-07-25 10:45 b****4 Rating: 5.0 Points 2024-06-28 10:40 b****4 Rating: 5.0 Points 2024-06-28 10:39 b****4 Rating: 5.0 Points 2024-05-30 13:17 w****1 Rating: 5.0 Points 2023-03-22 13:53 a****7 Rating: 5.0 Points 2023-02-15 09:12 h****4 Rating: 5.0 Points 2022-02-27 20:26 Jian****Wei Rating: 5.0 Points 2021-01-20 15:03 h****4 Rating: 5.0 Points 2020-08-04 09:27