I’m still speechless and overwhelmed by the speed and amount of donations I got to upgrade the DOTLAN EveMaps. It just took around 30 hours until I reached the 1.000€ goal. Woah, far faster then I ever imagined and expected. I didn’t even needed to make a lot of advertising myself. A lot of people spreaded the word for me on Twitter, Ingame, on Teamspeak channels, everywhere. Even CCP did some promotion for DOTLAN EveMaps on their Facebook page.
And what then happens is called the “Slashdot effect“. You get a lot of visitors in a very short amount of time. Since I didn’t configured the ‘max open file‘ value, the webserver wasn’t able deliver more content (internal error 500) even the server load was far from being critical. The default value wasn’t just good enough. Well you’ll always learn from your mistakes.
The downside: This night the server crashed again with this nice kernel panic. I’ve the strong feeling it might be related to bad memory, which should hopefully be solved with the new hardware and memory.
Lessons learned:
- be patient and wait until the hardware gets delivered (including registered ecc ram)
- adding watchdog devices/daemon. (If the server itself doesn’t ping the watchdog device via daemon, the server gets rebooted automatically)
- setting kernel.panic (sysctl.conf) to reboot after kernel panic
- setting fs.file-max (sysctl.conf), worker_rlimit_nofile and worker_connections (nginx.conf) to an appropriate value to handle enough file descriptors. Every http session, keep alive connection, reverse proxy connection or file access (static content) needs a file descriptor.