mysql service fails to start/hangs up - timeout (U

Question

I set up my first Ubuntu Server with Ubuntu 16.04, nginx, php7.0, MariaDB, nextcloud and external DynDNS using this tutorial here: Install Nextcloud 9 on Ubuntu 16.04

Everything worked fine but since I restarted the server the next day, nextcloud just shows me a blank page. After clicking through all logs of nginx, MariaDB and nextcloud, I found out that the mysql service just doesn't start. So run service mysql start and everything worked fine again (calling nextcloud from server as well as other workstations). I just wondered that the terminal didn't "close" the line. Like it was still working on the command. After about 5 minutes, the line "closes" and the following message appears:

"Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details."

Then the clients again just get a blank page in nextcloud. When I run the command and close the terminal immediately, clients get the access as well but still loses it after 5 minutes.

I tried backing up the nextcloud, sql and run apt-get purge --auto-remove mariadb-server. Then again run the MariaDB installation steps out of the tutorial with importing the backup sql instead of creating a new one. Didn't change everything.

Next try was update-rc.d mysql defaults and update-rc.d mysql enable. But after a restart just the blank page again. Access is only possible for 5 minutes by starting the service manual.

I also tried the BUM - BootUpManager but the service seems to be enabled. I saw you can start services out oft it manually as well. So tried it with mysql and surprise: nextcloud available for 5 minutes while BUM just hangs up.

I found mariadb.com/kb/en/mariadb/starting-and-stopping-mariadb-automatically/ as well but tried nothing of it because it seems like there is something else really wrong.

root@s1:~# systemctl status mariadb.service:

\u25cf mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: 
  Drop-In: /etc/systemd/system/mariadb.service.d
           \u2514\u2500migrated-from-my.cnf-settings.conf
   Active: failed (Result: timeout) since Di 2016-12-06 14:52:51 CET; 55s ago
  Process: 3565 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WS
  Process: 3415 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR
  Process: 3409 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START
  Process: 3405 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/ru
 Main PID: 3565 (code=exited, status=0/SUCCESS)

Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] /usr/sbin
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Sch
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: F
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: S
Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: W
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: S
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin
Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'.

root@s1:~# journalctl -xe:

Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Scheduler: Purging the queue. 0 events
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: FTS optimize thread exiting.
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: Starting shutdown...
Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer po
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: Shutdown completed; log sequence number 111890806
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin/mysqld: Shutdown complete
Dez 06 14:52:50 s1 audit[3648]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi
Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:29): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - 
Dez 06 14:52:50 s1 audit[3565]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi
Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:30): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - 
Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is failed.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'.
Dez 06 14:54:54 s1 x11vnc[2665]: 06/12/2016 14:54:54 cursor_noshape_updates_clients: 1
Dez 06 14:55:16 s1 ntpd[1244]: 46.4.1.155 local addr 192.168.178.50 -> <null>
Dez 06 14:57:30 s1 ntpd[1244]: 89.238.66.98 local addr 192.168.178.50 -> <null>

Content in /ect/init.d (if useful):

#!/bin/bash
#
### BEGIN INIT INFO
# Provides:          mysql
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Should-Start:      $network $named $time
# Should-Stop:       $network $named $time
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start and stop the mysql database server daemon
# Description:       Controls the main MariaDB database server daemon "mysqld"
#                    and its wrapper script "mysqld_safe".
### END INIT INFO
#
set -e
set -u
${DEBIAN_SCRIPT_DEBUG:+ set -v -x}

test -x /usr/sbin/mysqld || exit 0

. /lib/lsb/init-functions

SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)
CONF=/etc/mysql/my.cnf
MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"

# priority can be overriden and "-s" adds output to stderr
ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mysql -i"

# Safeguard (relative paths, core dumps..)
cd /
umask 077

# mysqladmin likes to read /root/.my.cnf. This is usually not what I want
# as many admins e.g. only store a password without a username there and
# so break my scripts.
export HOME=/etc/mysql/

# Source default config file.
[ -r /etc/default/mariadb ] && . /etc/default/mariadb

## Fetch a particular option from mysql's invocation.
#
# Usage: void mysqld_get_param option
mysqld_get_param() {
    /usr/sbin/mysqld --print-defaults \
        | tr " " "\n" \
        | grep -- "--$1" \
        | tail -n 1 \
        | cut -d= -f2
}

## Do some sanity checks before even trying to start mysqld.
sanity_checks() {
  # check for config file
  if [ ! -r /etc/mysql/my.cnf ]; then
    log_warning_msg "$0: WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz"
    echo                "WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz" | $ERR_LOGGER
  fi

  # check for diskspace shortage
  datadir=`mysqld_get_param datadir`
  if LC_ALL=C BLOCKSIZE= df --portability $datadir/. | tail -n 1 | awk '{ exit ($4>4096) }'; then
    log_failure_msg "$0: ERROR: The partition with $datadir is too full!"
    echo                "ERROR: The partition with $datadir is too full!" | $ERR_LOGGER
    exit 1
  fi
}

## Checks if there is a server running and if so if it is accessible.
#
# check_alive insists on a pingable server
# check_dead also fails if there is a lost mysqld in the process list
#
# Usage: boolean mysqld_status [check_alive|check_dead] [warn|nowarn]
mysqld_status () {
    ping_output=`$MYADMIN ping 2>&1`; ping_alive=$(( ! $? ))

    ps_alive=0
    pidfile=`mysqld_get_param pid-file`
    if [ -f "$pidfile" ] && ps `cat $pidfile` >/dev/null 2>&1; then ps_alive=1; fi

    if [ "$1" = "check_alive"  -a  $ping_alive = 1 ] ||
       [ "$1" = "check_dead"   -a  $ping_alive = 0  -a  $ps_alive = 0 ]; then
    return 0 # EXIT_SUCCESS
    else
    if [ "$2" = "warn" ]; then
        echo -e "$ps_alive processes alive and '$MYADMIN ping' resulted in\n$ping_output\n" | $ERR_LOGGER -p daemon.debug
    fi
    return 1 # EXIT_FAILURE
    fi
}

#
# main()
#

case "${1:-''}" in
  'start')
    sanity_checks;
    # Start daemon
    log_daemon_msg "Starting MariaDB database server" "mysqld"
    if mysqld_status check_alive nowarn; then
       log_progress_msg "already running"
       log_end_msg 0
    else
        # Could be removed during boot
        test -e /var/run/mysqld || install -m 755 -o mysql -g root -d /var/run/mysqld

        # Start MariaDB! 
        /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &

        # 6s was reported in #352070 to be too little
        for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do
                sleep 1
            if mysqld_status check_alive nowarn ; then break; fi
        log_progress_msg "."
        done
        if mysqld_status check_alive warn; then
                log_end_msg 0
            # Now start mysqlcheck or whatever the admin wants.
            output=$(/etc/mysql/debian-start)
        [ -n "$output" ] && log_action_msg "$output"
        else
            log_end_msg 1
        log_failure_msg "Please take a look at the syslog"
        fi
    fi
    ;;

  'stop')
    # * As a passwordless mysqladmin (e.g. via ~/.my.cnf) must be possible
    # at least for cron, we can rely on it here, too. (although we have 
    # to specify it explicit as e.g. sudo environments points to the normal
    # users home and not /root)
    log_daemon_msg "Stopping MariaDB database server" "mysqld"
    if ! mysqld_status check_dead nowarn; then
      set +e
      shutdown_out=`$MYADMIN shutdown 2>&1`; r=$?
      set -e
      if [ "$r" -ne 0 ]; then
        log_end_msg 1
        [ "$VERBOSE" != "no" ] && log_failure_msg "Error: $shutdown_out"
        log_daemon_msg "Killing MariaDB database server by signal" "mysqld"
        killall -15 mysqld
            server_down=
        for i in `seq 1 600`; do
              sleep 1
              if mysqld_status check_dead nowarn; then server_down=1; break; fi
            done
          if test -z "$server_down"; then killall -9 mysqld; fi
      fi
        fi

        if ! mysqld_status check_dead warn; then
      log_end_msg 1
      log_failure_msg "Please stop MariaDB manually and read /usr/share/doc/mariadb-server-10.1/README.Debian.gz!"
      exit -1
    else
      log_end_msg 0
        fi
    ;;

  'restart')
    set +e; $SELF stop; set -e
    $SELF start 
    ;;

  'reload'|'force-reload')
    log_daemon_msg "Reloading MariaDB database server" "mysqld"
    $MYADMIN reload
    log_end_msg 0
    ;;

  'status')
    if mysqld_status check_alive nowarn; then
      log_action_msg "$($MYADMIN version)"
    else
      log_action_msg "MariaDB is stopped."
      exit 3
    fi
    ;;

  'bootstrap')
    # Bootstrap the cluster, start the first node
    # that initiates the cluster
    log_daemon_msg "Bootstrapping the cluster" "mysqld"
    $SELF start "${@:2}" --wsrep-new-cluster
    ;;

  *)
    echo "Usage: $SELF start|stop|restart|reload|force-reload|status|bootstrap"
    exit 1
    ;;
esac

Unfortunately, Google can't help me. I tried to explain as much as I can maybe this helps you in helping me. Thanks a lot!

Answer 1

Long question for nothing... Never heard of AppArmor but it was the reasen. The answer here fixed it. Don't care about apparmor's ERROR the profile wouldn't exist.

sudo aa-status shows you what apparmor is doing; what actually has an enforced policy, versus what's just set to complain.

sudo apt-get install apparmor-utils adds a few commands that make the apparmor profiles easier to deal with, such as...

sudo aa-complain /usr/sbin/mysqld turns the profile from "enforce" to complain. (aa-enforce turns it back.)

Once that's done, sudo service apparmor reload restarts apparmor, and voila... sudo /etc/init.d/mysql start works, and the server stays up.

Answer 2

Moving mysqld to "complain" group was not enough in my case (MariaDB 10.1.21 running on Ubuntu 16.04). I had to fully disable apparmor for mysqld:

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ 
sudo service apparmor reload
sudo service mysql restart

Now everything works fine.

Answer 3

Run the following commands:

sudo dpkg --configure -a
sudo service mysql start

Answer 4

Please note that since 10.1.10, MariaDB uses systemd to start the service. If you have tried MYSQLD_STARTUP_TIMEOUT and it has not worked, you are probably using this or a later version. The /etc/init.d/mysql script is no longer used, so MYSQLD_STARTUP_TIMEOUT has no effect.

You need to find your mariadb.service file. In our case, it did not contain a timeout so the MariaDB default was being used. Just add:

TimeoutStartSec = 0

In the [Service] section, and it will never time out.

It would be a good idea to create your own config file containing this so it doesn't get overwritten by later re-installs.

On ubuntu 18.04, you will fine this file in

/lib/systemd/system/mariadb.service

Put your own file in

/etc/systemd/system/mariadb.service.d

Remember to run systemctl daemon-reload after adding the timeout somewhere (and maybe check /var/log/syslog to see if the reload was successful), otherwise your time out will be ignored.

Answer 5

This last option worked for me (from quazgar). I have Ubuntu 18.10 installed with MariaDB 10.3.13:

$ echo "/usr/sbin/mysqld { }" > /etc/apparmor.d/usr.sbin.mysqld
$ apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld
$ systemctl restart mariadb

I had to use "sudo su" for it to work though.

Answer 6

In case you are bitten by this bug, the solution is given as a suggestion in the bug report:

1. echo "/usr/sbin/mysqld { }" > /etc/apparmor.d/usr.sbin.mysqld
2. apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld
3. systemctl restart mariadb

Background

If you previously had MySQL installed, it activated an AppArmor profile which is incompatible with MariaDB. apt-get remove --purge only removes the profile, but does not deactivate/unload it. Only manually unloading it lets MariaDB work unhindered by AppArmor.