Mysterious lack of disk space
Posted on: 25 August 2009
An interesting problem for today. I was trying to fix a fairly trivial bug in one of our websites, but when I try to load a page from my webserver I get a Drupal error:
user warning: Can't create/write to file '/tmp/#sql_1071_0.MYI' (Errcode: 28) query: SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.vid = 39 ORDER BY v.weight, t.weight, t.name in /var/www/seo/modules/taxonomy/taxonomy.module on line 617.
Errcode 28 means I'm out of disk space, but a quick df shows plenty:
Filesystem Size Used Avail Use% Mounted on /dev/sda1 3.8G 3.1G 493M 87% / varrun 252M 188K 252M 1% /var/run varlock 252M 0 252M 0% /var/lock udev 252M 52K 252M 1% /dev devshm 252M 0 252M 0% /dev/shm /dev/sdb1 4.0G 2.2G 1.7G 57% /var/www
Trying to copy even a small file on the file system also gives a "No space left on device" error. A little more digging around the error message suggests checking the inode usage too, with df -i:
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 247008 247008 0 100% / varrun 64430 55 64375 1% /var/run varlock 64430 2 64428 1% /var/lock udev 64430 2777 61653 5% /dev devshm 64430 1 64429 1% /dev/shm /dev/sdb1 524288 92549 431739 18% /var/www
Ahah! But what could possibly have used quarter of a million inodes on my system disk? A little more digging around finds the culprit:
mark@dev:/root$ ls -1R | wc -l 181234
I'm running There are rather a lot of empty files with names like cron.php.16966. Sadly, too many to delete with a simple rm cron*, but a quick shell script soon tidied them up:
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 247008 65780 181228 27% /
Much better!
Strange though, as I thought my cron jobs were configured to not generate any output at all:
0 * * * * wget http://mywebsite/cron.php >/dev/null 2>&1
Turns out I was missing some rather important parameters off the wget command to suppress output:
0 * * * * wget -O http://mywebsite/cron.php -q >/dev/null 2>&1
Now all my cron jobs are running nice and quietly, and I can get back to some real work!