Record the process of a Linux machine cleaning up space

Posted Jun 28, 20202 min read

In the project, we need to change the configuration of Nginx, so we log in to the Nginx machine through the springboard. The result saw this scene:
image.png
Exciting, the server space is gone. Let's solve this problem first, otherwise it can't be written, and the server can't run normally.

analysis:
This machine is only responsible for running Nginx, and the space occupied is largely due to too much log generation.

According to the analysis ideas, I first look at the status of disk space usage:

Command:df -h

image.png
There is indeed a block with a utilization rate of up to 100%, but I don t know about this block, I don t know which block is so full, so I went to the root directory to see which directory is so large:

Command:du -sh /*

image.png
Found out, it was originally home, by viewing all the files in the current directory according to size, and finally located to the Nginx directory:

Instruction:du -s * | sort -nr

image.png
After confirming that error.log can be deleted, delete the file:

Command:rm error.log(Do not use rm -rf casually)

image.png
It turned out that after cleaning, the system still indicated that there was not enough space, and the file was clearly clear. What happened? I took another look and confirmed that the correct file was deleted:

Instruction:du -sh * | sort -nr

image.png
It can be seen that the error log that occupied 59G of space just now has been deleted, but there is still insufficient space. At this time, I thought, is it because nginx is running, the error.log file is being occupied, and the deletion operation is saved, and can only be deleted after the nginx process ends. So view the information in the delete queue:

Command:lsof | grep deleted

image.png
As expected, these files actually exist, so next time you need to temporarily close the nginx release process to allow the system to perform these deletion tasks, but considering that someone may be using nginx, in order to minimize the impact, so use the response speed more Speed up the reopen operation and restart nginx(there is only one restart command, which is equal to stop+start, which is definitely faster than entering stop and then starting manually, and the impact is smaller):

Instruction:nginx -s reopen

When I restarted, I obviously felt a bit stuck, it seems that the deletion task has been performed, 59G files have been cleaned up, check:

Command:Still lsof | grep deleted

image.png
It can be seen that the file has been cleaned up.
Check with the du command, you can see that the home directory has successfully lost 59G, and the system does not prompt that the space is full~:
image.png

to sum up:
Pay attention to cleaning up files, use rm -rf carefully.
After cleaning, it is still not available, check to see if it is occupied.