When to Scale: Judging Resource Headroom via Trend Data

The two extremes to avoid in scaling decisions are: waiting until 100% capacity (when incidents have already occurred), and buying based on intuition (which wastes resources). The rational approach is to monitor trends and remaining time, rather than focusing solely on the current percentage.

Core Concept: Calculate “How Long Can We Hold?”

The current usage rate itself has limited significance. The truly useful metric is:

Based on the current growth rate, how many days remain until the critical threshold is reached?

The same 70% usage implies completely different urgency levels depending on the growth speed.

How to Obtain Trend Data

① Periodic Recording (Simplest Method)

#!/bin/bash
# /root/record-usage.sh
DATE=$(date +%F)
echo "$DATE $(df -h / | awk 'NR==2 {print $5}') $(free -m | awk 'NR==2 {print $3}')" 
  >> /var/log/usage-trend.log

0 9 * * * /root/record-usage.sh

# View trends
cat /var/log/usage-trend.log

Example Output:
2026-10-01 65% 2400
2026-10-05 68% 2450
2026-10-08 71% 2510
→ Increased by 6% in 3 days, approx. 2% per day. ~10 days left to reach 90%

② Use Historical Data from Monitoring Tools

If you have a monitoring system, viewing historical curves is more intuitive. The slope of the curve is more important than the absolute value.

Evaluating Different Resources

# Check disk and inode together
df -h | grep -vE 'tmpfs|overlay'
df -i | grep -vE 'tmpfs|overlay'

# Memory (check available; true shortage is indicated when accompanied by swap usage)
free -m

# Database connection history peak (highly referenceable)
mysql -e "SHOW STATUS LIKE 'Max_used_connections'; SHOW VARIABLES LIKE 'max_connections';"

Confirm Growth is Normal First

When fast growth is detected, first investigate what is growing before deciding to scale—some growth is abnormal, and cleaning it up may eliminate the need for scaling.

# Identify sources of growth
du -sh /var/* 2>/dev/null | sort -rh | head -10

# Compare with records from a few days ago (if available)
# Or directly check recently modified large files
find / -type f -size +100M -mtime -7 -exec ls -lh {} ; 2>/dev/null | head

“Investigate source before scaling” saves unnecessary costs. Scaling due to unrotated logs only delays the problem.

Choosing Scaling Methods

# After disk expansion (cloud disk), partition and filesystem must also be extended
# Specific steps depend on partitioning method; always take a snapshot backup first

# Check current status
lsblk
df -h

Disk expansion involves partition operations; snapshots are mandatory. While cloud disk expansion itself is safe, subsequent steps to extend partitions carry risks of error.

Prevention: Set Tiered Alerts

Recommended Alert Thresholds:
  Disk 80%  → Reminder (Start paying attention, check growth source)
  Disk 90%  → Warning (Handle ASAP)
  Disk 95%  → Critical (Immediate action)

  Memory available < 20% sustained → Warning
  Connections > 80% of max_connections → Warning

#!/bin/bash
# Tiered alert example
df -h | awk 'NR>1 {gsub("%","");
  if ($5 >= 95) print "Critical: "$6" "$5"%";
  else if ($5 >= 90) print "Warning: "$6" "$5"%";
  else if ($5 >= 80) print "Reminder: "$6" "$5""%"}'

Decision Tree

Should I Scale?
├─ Look at trends, not just current values
│  └─ Calculate "days remaining to threshold" based on growth rate
├─ Investigate growth source first
│  ├─ Normal business growth → Scale up
│  └─ Abnormal (logs/backups/temp files) → Clean up, no scaling needed
├─ Choose scaling method
│  ├─ Disk → Cloud disk expansion (snapshot first)
│  ├─ Memory/CPU → Upgrade instance (usually requires restart, schedule window)
│  └─ High overall load → Add nodes + Load Balancer
└─ Prevention
   └─ Tiered alerts (80% Reminder / 90% Warning / 95% Critical)

Scenario Analysis: Why Every Expansion Feels Like “It Should Have Been Done Earlier”

Phenomenon: A server’s resources repeatedly approach their limits.
Scaling is only performed after performance degrades or issues arise. Post-mortem reviews often reveal comments like “it should have been scaled earlier,” yet no one could provide justification beforehand.

1. Define the basis for “should have been done earlier” →
   It is not the current water level, but the time required to reach the limit. Currently at 70% and “will hit 100% in a week” are two different things.
2. Accumulate trend data →
   Regularly collect and retain key metrics; at least several consecutive data points are needed to calculate velocity.
3. Estimate remaining time based on growth rate →
   Turn “how long is left” into numbers that can be documented, rather than relying on experience.
4. Set two warning lines →
   An earlier line for initial assessment, and a closer line for mandatory action.
5. Consider periodic events →
   When business has clear peaks and valleys, judge headroom based on peak values, not averages.
6. Verify effects after scaling →
   Confirm that new resources are actually utilized; lack of improvement usually indicates an unresolved bottleneck elsewhere.

# Retain historical data points to calculate trends (a single snapshot cannot show growth rate)
echo "$(date +%F) $(df -h / | awk 'NR==2{print $5}')" >> disk-trend.txt
cat disk-trend.txt

# Disk growth trend + estimate remaining days based on current speed
df -h / | awk 'NR==2 {print "Used:", $5, " Available:", $4}'

# Memory: Distinguish between actual usage and cache
free -m
# Focus on the 'available' column

# Load: Only meaningful when compared to core count
nproc
cat /proc/loadavg

Root Cause: Relying solely on current water levels without trend data.
“Currently sufficient” and “will be insufficient next week” look identical in a single data point; only continuous data distinguishes them.

Two Key Takeaways:

• The basis for scaling is not “current amount,” but “time remaining at this speed.”
  The latter requires continuous data points, making historical retention a prerequisite.
• Effects must be verified after scaling.
  If added resources do not yield improvement, the bottleneck likely lies outside the scaled layer.

8 Common Pitfalls

1. Focusing only on current percentage: Ignoring growth rate.
2. Scaling without investigating sources: Potentially wasting money.
3. Checking df -h but ignoring df -i: Inodes can also be exhausted.
4. Waiting until 100% to act: Incidents have already occurred.
5. Skipping snapshots before disk expansion: Partition operations carry risks.
6. Upgrading without scheduling a maintenance window: Restart is usually required.
7. Lack of historical records: Trends cannot be calculated.
8. Having only one alert threshold: Tiered alerts are more effective.

Real Customer Feedback (From Cloud Market Reviews and Consultations for Similar Products)

The following excerpts are taken from real buyer reviews and consultations for similar O&M services on Alibaba Cloud / Tencent Cloud / Huawei Cloud Cloud Markets (providers include our store and peers; provided for reference only, not fictitious tickets from our site):

1****4 Rating: 5.0 2023-05-08 11:53 1****3 Rating: 5.0 2022-09-22 22:01 r****u Rating: 5.0 2020-07-07 15:05

b****4 Rating: 5.0 2025-06-23 16:57 b****4 Rating: 5.0 2025-06-23 16:56 Jian****Wei Rating: 5.0 2021-01-20 15:03 Technical skills are awesome, speed is fast! It turned out to be a trojan causing Baidu global hijacking!!!!!1 f****u Rating: 5.0 2021-01-02 20:43 Thanks for assisting in maintaining Yuncai’s normal operation. 1****1 Rating: 5.0 2020-11-20 10:03 Consistently good service as always. 1****1 Rating: 5.0 2020-11-17 09:13 Thank you for the consistently good service. 1****1 Rating: 5.0 2020-09-25 15:42 Many thanks. 1****1 Rating: 5.0 2020-08-24 15:34 Sincere thanks to your company’s technical staff. 1****1 Rating: 5.0 2020-07-24 09:49 Many thanks!

d****9 Rating: 5.0 2023-02-15 09:11 Yong****Ying Rating: 5.0 2021-03-29 11:04