Mysterious lack of disk space

Posted on: 25 August 2009

An interesting problem for today. I was trying to fix a fairly trivial bug in one of our websites, but when I try to load a page from my webserver I get a Drupal error:

user warning: Can't create/write to file '/tmp/#sql_1071_0.MYI' (Errcode: 28) 
query: SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid 
INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.vid = 39 
ORDER BY v.weight, t.weight, 
in /var/www/seo/modules/taxonomy/taxonomy.module on line 617.

Errcode 28 means I'm out of disk space, but a quick df shows plenty:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.8G  3.1G  493M  87% /
varrun                252M  188K  252M   1% /var/run
varlock               252M     0  252M   0% /var/lock
udev                  252M   52K  252M   1% /dev
devshm                252M     0  252M   0% /dev/shm
/dev/sdb1             4.0G  2.2G  1.7G  57% /var/www

Trying to copy even a small file on the file system also gives a "No space left on device" error. A little more digging around the error message suggests checking the inode usage too, with df -i:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             247008  247008       0  100% /
varrun                 64430      55   64375    1% /var/run
varlock                64430       2   64428    1% /var/lock
udev                   64430    2777   61653    5% /dev
devshm                 64430       1   64429    1% /dev/shm
/dev/sdb1             524288   92549  431739   18% /var/www

Ahah! But what could possibly have used quarter of a million inodes on my system disk? A little more digging around finds the culprit:

mark@dev:/root$ ls -1R | wc -l

I'm running There are rather a lot of empty files with names like cron.php.16966. Sadly, too many to delete with a simple rm cron*, but a quick shell script soon tidied them up:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             247008   65780  181228   27% /

Much better!

Strange though, as I thought my cron jobs were configured to not generate any output at all:

0 * * * * wget http://mywebsite/cron.php >/dev/null 2>&1

Turns out I was missing some rather important parameters off the wget command to suppress output:

0 * * * * wget -O http://mywebsite/cron.php -q >/dev/null 2>&1

Now all my cron jobs are running nice and quietly, and I can get back to some real work!