Just a quick wrap up from the last 24h. Yesterday at 10am GMT one of my disks (lvm slice) was running full. Of course programs and databases don’t like when they don’t have any space left to write their temp data (sessions, updates, etc).
The first problem was solved pretty quickly. I just had to offline evemaps, extend the lvm slice and start evemaps again and do some checks and repair some of the database tables … Problemo solved …
Sometimes problems could be that easy (if they would be detected early enough), but not in this case: a different background job found some corrupted caches and didn’t had anything better to do then spam bullshit into the logfile, etc that fast that the drive was full again in the middle of the night.
Anyway … problem was finally fixed at 6am GMT this morning. I’ll keep an eye on the current situation.