Error restarting Openshift MySQL cartridge

Question

I have a non-scalable OpenShift application with Tomcat 7 and MySQL 5.5 and this application has only one small Java application running on it. This application was running fine for a couple of months but since last week it stopped working. When I try to restart the application using the OpenShift dashboard I get the following error message:

Starting MySQL 5.5 cartridge
MySQL server failed to start:
140305 07:01:47 mysqld_safe Logging to '/var/lib/openshift/53104e375973cad496000022/mysql//log/mysql_error.log'.
140305 07:01:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/53104e375973cad496000022/mysql/data/
140305 07:01:52 mysqld_safe mysqld from pid file /var/lib/openshift/53104e375973cad496000022/mysql/pid/mysql.pid ended
140305 07:01:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/53104e375973cad496000022/mysql/data/
140305  7:01:49 [Note] Plugin 'FEDERATED' is disabled.
140305  7:01:49 InnoDB: The InnoDB memory heap is disabled
140305  7:01:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140305  7:01:49 InnoDB: Compressed tables use zlib 1.2.3
140305  7:01:49 InnoDB: Using Linux native AIO
140305  7:01:49  InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
140305  7:01:52  InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
140305  7:01:52 InnoDB: Fatal error: cannot initialize AIO sub-system
140305  7:01:52 [ERROR] Plugin 'InnoDB' init function returned error.
140305  7:01:52 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140305  7:01:52 [ERROR] Unknown/unsupported storage engine: InnoDB
140305  7:01:52 [ERROR] Aborting

140305  7:01:52 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Shutdown complete

140305 07:01:52 mysqld_safe mysqld from pid file /var/lib/openshift/53104e375973cad496000022/mysql/pid/mysql.pid ended
Failed to execute: 'control restart' for /var/lib/openshift/53104e375973cad496000022/mysql

Apparently the MySQL cartridge is failing to restart somehow, but I didn't change anything in my Java application or the gear configuration.

Does anybody know why this is happening? I don't want to simply remove MySQL and add it again because this problem may happen again in the future if I don't identify the root cause.

Thanks in advance.

Answer 1

I would suggest checking your quota to see if you've reached your limit. Try ssh'ing into your gear and running the following quota -s. If you've reached your quota then you can use the rhc tidy command on your local machine to clean up the gear.

EDIT:

After a little more research I would say that you were affected by this bug https://bugzilla.redhat.com/show_bug.cgi?id=1061926. In the last few days we experienced some outage time with some residual effects. Rest assured we are actively working getting this fully resolved.

Answer 2

Having a similar problem with restarting mysql on http://fundraiser-aapk.rhcloud.com/ Do I need to file a bug/support request someplace?

BTW My quota output is as follows.

 Filesystem  blocks   quota   limit   grace   files   quota   limit   grace

/dev/mapper/EBSStore01-user_home01

               500M       0   1024M           38899       0   80000