All of a sudden I had started receiving DOWN alerts from Pingdom in the middle of the night almost every week. Each time I checked the source of the issue I’ve seen that MySQL server was crashing or stopping somehow causing the Pingdom alerts. I ignored it for the first time, as I do not host important web sites but when it happened for the second time exactly 1 week after, now I was curious. But the same problem, happening on a Sunday night, one week another? Looks like a pattern, so it might be a cronjob!
No it wasn’t. I checked all the scheduled jobs but there were nothing related. Then as MySQL hadn’t enabled log files by default (!) I’ve modified the config file to add the log file line:
$ emacs -nw /etc/mysql/my.cnf log = /var/log/mysql/mysql.log
I did restart MySQL server and started to wait until next week, again, precisely at around 5:30am on Sunday. Checking the log files I saw the following messages:
InnoDB: Completed initialization of buffer pool InnoDB: Fatal error: cannot allocate memory for the buffer pool [ERROR] Plugin 'InnoDB' init function returned error. [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. [ERROR] Unknown/unsupported storage engine: InnoDB [ERROR] Aborting
Googling showed me I was not the only one with this problem as an Amazon EC2 user. Applying the steps in this post on SO: http://stackoverflow.com/questions/10284532/amazon-ec2-mysql-aborting-start-because-innodb-mmap-x-bytes-failed-errno-12 solved the issue. It’s been 3 weeks now and the crashes have disappeared so far.