If you’ve may noticed, the evemaps webserver is crashing every now and then. It looks like some background scripts, which are keeping the database up to date, are crashing and killing the server at the same time, leaving the server in an unmanageable state. No logfile outputs, no console output, nothing. The current solution is to destroy the virtual machine instance and start the server again, which is … not acceptable.
I’m thinking about installing a clean virtual machine and move evemaps to this or rewrite the maintenance scripts or compiling php from scratch. Right now I’m still searching for the root cause, which is hard to find without any usable logfiles or debug output or beeing able to reproduce the crash.
At least I’ve setup a monitoring system on a different computer which notifies my via jabber/email and later text message to my mobile, if the server crashed again. Maybe I’ll add a auto-restart script on the xen dom0.
Update
So far the server keeps crashing every day without any logfiles or error messages. After several tries to find the crash reason, webserver/php reinstall, code rewriting, I decided to move evemaps to their own single, fresh installed virtual server instance. Lets hope this helps. Otherwise I might consider compiling php from scratch rather then using a precompiled debian packge. I’ll also try to get to the datacenter and do some memtests with the current server setup this weekends.
Update 2
I’ll be in the datacenter around 19:30 GMT to exchange ram modules and do some memory checks (just in case). DOTLAN EveMaps will be down for around 30 to 45 minutes.