Mysql won't start - ibdata1 corrupt? - operati

Question

Server shutdown from power failure.
Mysql will not start now.
Disk is not full. Syslog is below

Oct 11 15:03:31 joe mysqld_safe[24757]: started
Oct 11 15:03:31 joe mysqld[24760]: 101011 15:03:31  InnoDB: Operating system error number 13 in a file operation.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: The error means mysqld does not have the access rights to
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: the directory.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: File name ./ibdata1
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: File operation call: 'create'.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: Cannot continue operation.

Answer 1

If you use SEL Linux

Intall semanage

yum whatprovides /usr/sbin/semanage you get policycoreutils-python-2.5-22.el7.x86_64

See mysqld security context

After installation yum install policycoreutils-python you can just look what different security context mysqld has.

semanage fcontext -l | grep mysqld
/etc/mysql(/.*)?                       all files    system_u:object_r:mysqld_etc_t:s0
/etc/my\.cnf\.d(/.*)?                  all files    system_u:object_r:mysqld_etc_t:s0
/var/log/mysql.*                       regular file system_u:object_r:mysqld_log_t:s0
/var/lib/mysql(-files|-keyring)?(/.*)? all files    system_u:object_r:mysqld_db_t:s0
/var/run/mysqld(/.*)?                  all files    system_u:object_r:mysqld_var_run_t:s0
/var/log/mariadb(/.*)?                 all file     system_u:object_r:mysqld_log_t:s0
/var/run/mariadb(/.*)?                 all files    system_u:object_r:mysqld_var_run_t:s0
/usr/sbin/mysqld(-max)?                regular file system_u:object_r:mysqld_exec_t:s0
/var/run/mysqld/mysqlmanager.*         regular file system_u:object_r:mysqlmanagerd_var_run_t:s0
/usr/lib/systemd/system/mysqld.*       regular file system_u:object_r:mysqld_unit_file_t:s0
/usr/lib/systemd/system/mariadb.*      regular file system_u:object_r:mysqld_unit_file_t:s0
/etc/my\.cnf                           regular file system_u:object_r:mysqld_etc_t:s0
/root/\.my\.cnf                        regular file system_u:object_r:mysqld_home_t:s0
/usr/sbin/ndbd                         regular file system_u:object_r:mysqld_exec_t:s0
/usr/libexec/mysqld                    regular file system_u:object_r:mysqld_exec_t:s0
/usr/bin/mysqld_safe                   regular file system_u:object_r:mysqld_safe_exec_t:s0
/usr/bin/mysql_upgrade                 regular file system_u:object_r:mysqld_exec_t:s0
/etc/rc\.d/init\.d/mysqld              regular file system_u:object_r:mysqld_initrc_exec_t:s0
/var/lib/mysql/mysql\.sock             socket       system_u:object_r:mysqld_var_run_t:s0
/usr/libexec/mysqld_safe-scl-helper    regular file system_u:object_r:mysqld_safe_exec_t:s0
/home/[^/]+/\.my\.cnf                  regular file unconfined_u:object_r:mysqld_home_t:s0

Here you see all context for mysqld a short list with explanation

1. mysqld_etc_t - config files
2. mysqld_db_t - data db files
3. mysqld_log_t - log files
4. mysqld_exec_t - execution files

So if you have the wrong security context on your files you get a permission denied (error 13)

Solution

chcon -R -u system_u -t mysqld_db_t  /var/lib/mysql

But check the "normal" permissions, too. I had this problem with centos. You have to systemctl restart mysql for the changes.

Answer 2

When this popped up for me, I found the answer in the /etc/mysql/my.cnf configuration file. The datadir line did not point to the /var/lib/mysql directory (where the databases are). Once I put this path in, the server restarted no problem.

Answer 3

If you are using ubuntu or apparmor you should permit this change in apparmor.

Edit /etc/apparmor.d/usr.sbin.mysqld and change /var/lib/mysql with the new DATADIR.

It should work.

Answer 4

In short, (especially on RHEL/CentOS/Fedora) try

getenforce

if it replies with Enforcing you have SELinux up and running. Temporarily deactivate it with setenforce 0 and see if MariaDB starts now! Rather common, especially on RHEL/CentOS/Fedora.

There's more about this further down, as well as in this official article.

In general

There are more things in a UNIX environment that might prevent file access, than just user access rights.

• Security modules like SELinux (see above) or AppArmor (as Dan mentioned) could disallow it
• Access Control Lists (ACL) could be specifically set, for the required files/directories
• Any of the parent folders could be owned by another user, and have no x (="dir access") set for others

Additionally there could be other unexpected factors, like ...

• The mysql datadir being set to a place, where mysql doesn't have permissions (see /etc/my.cnf)
• Mysql could (strangely) be running as a different user, or the file could be simply owned by someone else

Just to mention a view things off the top of my head (feel free to edit/add to this answer btw).

In the case, SELinux is "the problem"

For a permanent solution, you could try to restore the appropriate security context, ...

restorecon -R /var/lib/mysql/

... or just deactivate SELinux (but think about this one a little bit before doing so), by editing the config (typically in /etc/selinux/config) and setting SELINUX=disabled as suggested in following article.

• Here the official help page from mariadb.com: What to do if MariaDB doesn't start
• And here something from redhat.com: MariaDB Changing Database Location

Obviously those are applicable to MySQL just the same way.

Answer 5

I had the same problem. Did alot of research and found out this solution. You need to run this command on ibdata1 sudo shadowprotect -u root | root

I dont know what this does.. but it worked for me.

Good luck.

Answer 6

The file is not corrupt. You can find out the source of these errors with 'perror'. i.e.

toaster:~ morgo$ perror 13
OS error code  13:  Permission denied

InnoDB has corruption detection (page checksums) and would happily tell you if that were the problem.

Either the directory permissions have changed, or your my.cnf file has been hosed, and it's trying to recreate data files somewhere else.