How to Safely Expand Full Disks: Handling Linux Partition Mounting and df/du Inconsistencies

When a server reports that the disk is full, websites become unwritable, and databases throw errors. The difficulty with these issues lies not in “expansion,” but in two easily overlooked aspects:

The first is “df shows the disk is full, but du statistics sum up to much less than df”—it’s unclear where the space went, making it hard to act safely; the second is the cloud console shows expansion to 200G, but df still reports the original 40G—money was spent, but the space didn’t change.

First, Identify Which Type of “Full” You Are Facing

# System level
No space left on device

# MySQL
Error writing file '/tmp/MLXXXX' (Errcode: 28 - No space left on device)

# nginx
write() to "/var/log/nginx/access.log" failed (28: No space left on device)

# Symptoms of inode exhaustion -- space remains, but still reports No space
df -h shows ample remaining space

Troubleshooting Commands

# ① Check both space and inodes
df -h
df -i

# ② Find directories consuming space
du -sh /var/* 2>/dev/null | sort -rh | head -10
du -ah / 2>/dev/null | sort -rh | head -20

# ③ Crucial: Files deleted but space not released (must check when df is full but du is not)
lsof | grep deleted
# The correct handling is not rm, but clearing file content while keeping the handle
: > /var/log/nginx/access.log

# ④ View mount points and filesystem types -- determines which command to use for expansion
lsblk -f
df -T
mount | grep ' / '

# ⑤ Check if mounted as read-only
mount | grep -w ro

Why Do df and du Inconsistencies Occur?

This is the most confusing aspect. In Linux, rming a file only removes the directory entry; if a process still has the file open, the disk blocks it occupies will not be released.

Typical scenario: A log file grows to tens of GB, and an operator directly runs rm access.log, then finds that df hasn’t changed at all—because the nginx process still holds the handle to that deleted file.

There are two correct approaches:

# Approach 1: Clear file content (Recommended, does not affect processes currently writing)
: > /var/log/nginx/access.log

# Approach 2: Have the process holding the handle reopen the file
# For nginx, use reload
nginx -s reload

# If already deleted via rm -- find the holder and handle it
lsof | grep deleted

Expansion Steps

Scenario A: Cloud Disk Already Expanded in Console, Expand Within System

# ① Extend partition (example using the 1st partition of /dev/vda)
growpart /dev/vda 1

# ② Extend filesystem -- ext4 and xfs commands differ, check type with df -T first
resize2fs /dev/vda1        # ext4: follows device name
xfs_growfs /               # xfs: follows mount point, note it's not the device name

# ③ Confirm result
df -h

Expanding only the cloud disk without expanding the filesystem is the most frequent mistake in these issues: the console shows 200G, but df inside the system still shows 40G because the operating system hasn’t perceived the disk size increase.

Scenario B: Adding a New Data Disk

# ① Confirm the new disk is recognized
fdisk -l

# ② Partition (for disks larger than 2T, you must use parted / GPT)
parted /dev/vdb mklabel gpt
parted /dev/vdb mkpart primary ext4 0% 100%

# ③ Format
mkfs.ext4 /dev/vdb1

# ④ Mount
mkdir -p /data
mount /dev/vdb1 /data

# ⑤ Auto-mount on boot -- use UUID, do not write device names
blkid /dev/vdb1
echo 'UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /data ext4 defaults 0 0' >> /etc/fstab

# ⑥ Must test first; syntax errors can prevent booting
mount -a

Decision Tree

Disk Full Warning
├─ df -h is full
│  ├─ du matches         → Clean logs / backups / binlog, or expand
│  └─ du does not match  → lsof | grep deleted → Clear file or restart process
├─ df -h has space, df -i is full → Inode exhaustion → Clean massive small files (sessions, cache fragments)
├─ Cloud disk expanded but system size unchanged → growpart + resize2fs / xfs_growfs
└─ Space sufficient but write fails → Check if mounted read-only or filesystem corrupted

Scenario Walkthrough: df Says Full, But du Can’t Find Large Files

Phenomenon: Monitoring alerts that a server’s root partition usage has reached 100%.
After logging in, you try to find large directories using standard methods to clean them up one by one, but find that the summed total is far less than the used space.

1. Confirm if it’s truly full or a statistical discrepancy → Filesystem level shows usage near limit, available is 0.
   This step rules out false alarms.
2. Summarize usage by directory, find mismatch → Total is tens of GB less than used amount, indicating space is “hidden” by statistical口径.
3. Consider files deleted but still held by processes → In Linux, deleting a file being written to does not immediately release space until the process holding it ends.
4. Find these deleted but still open files → Locate several large-volume deleted log files, held by a long-running service.
5. Confirm safe handling → Cannot kill the service directly; instead, notify the service to reopen log files to release handles.
6. Configure log rotation after space release → Space drops immediately, and log rotation is configured to prevent recurrence.

# Standard method to check df vs du inconsistency: find "deleted but still open" files
lsof +L1 | head -20
lsof | grep -i deleted | head -20

# See how much specific files occupy
lsof | grep -i deleted | awk '{print $7, $9}' | sort -rn | head

# Release log handles without stopping services (using common services as examples)
# Method 1: Notify process to reload (Recommended)
kill -USR1 ProcessPID

# Method 2: Truncate the handle, suitable for situations where restarting is not appropriate
# : > /proc/ProcessPID/fd/FileDescriptor

# Confirm space is released
df -h

Root Cause: Log files were directly deleted, but the process writing to them still holds the handle, preventing space release;
The filesystem shows full while directory statistics cannot find corresponding files, which is the reason.

Two key points to remember:

• If disk is full and the sum of du statistics is significantly less than df’s used amount, it is almost certainly deleted but occupied files.
  Directly searching for large directories wastes time; the first step should be checking occupied deleted files.
• Do not directly delete log files when cleaning.
  The correct approach is to have the service reopen the log (send signal to reload) or clear content using truncate.
  Deleting the file itself is ineffective when space is fully occupied.

8 Common Pitfalls

1. Expanding only the cloud disk without expanding the filesystem: Must run growpart + resize2fs, otherwise no expansion occurred.
2. Confusing ext4 and xfs commands: ext4 uses resize2fs /dev/vda1 (follows device), xfs uses xfs_growfs / (follows mount point); swapping them causes errors.
3. Writing device names in fstab: /dev/vdb1 may change after reboot, causing boot failure if not mounted; always use UUID.
4. Not running mount -a after modifying fstab: Syntax errors will lead directly to rescue mode on next boot.
5. Directly rming logs being written: Space will not be released; use : > file to clear instead.
6. Checking only df -h but not df -i: Inode exhaustion looks identical to disk full, but cleanup methods are completely different.
7. Not backing up before partitioning: Partitioning data disks carries high risk; always take snapshots or backups first.
8. Long-term retention of MySQL binlogs: They silently consume tens of GB; configure automatic expiration cleanup policies.

Real Customer Feedback (From Cloud Market Reviews and Consultations for Similar Products)

The following excerpts are from real buyer reviews and consultations for similar O&M services on Alibaba Cloud / Tencent Cloud / Huawei Cloud marketplaces (service providers include our store and peers, provided for reference only, not fictional tickets from this site):

b****4 Rating: 5.0 Jun 23, 2025 16:57 b****4 Rating: 5.0 Jun 23, 2025 16:56 Jian****Wei Rating: 5.0 Jan 20, 2021 15:03 Technology is awesome, speed is fast! It turned out to be a trojan causing global Baidu hijacking!!!!!! f****u Rating: 5.0 Jan 02, 2021 20:43 Thanks for assisting in maintaining Yuncai normal operation. 1****1 Rating: 5.0 Nov 20, 2020 10:03 Consistent good service as always 1****1 Rating: 5.0 Nov 17, 2020 09:13 Thanks for the consistent good service. 1****1 Rating: 5.0 Sep 25, 2020 15:42 Many thanks. 1****1 Rating: 5.0 Aug 24, 2020 15:34 Many thanks to your company’s technical staff 1****1 Rating: 5.0 Jul 24, 2020 09:49 Many thanks! 1

d****9 Rating: 5.0 Feb 15, 2023 09:11 Yong****Ying Rating: 5.0 Mar 29, 2021 11:04

1****4 Rating: 5.0 May 08, 2023 11:53 1****3 Rating: 5.0 Sep 22, 2022 22:01 r****u Rating: 5.0 Jul 07, 2020 15:05