Apache server keeps crashing, “caught SIGTERM, shu

Question

This just started happening three weeks or so ago. The content of my website hasn't changed, it's just a phpBB forum using MySQL as a backend.

Nothing has changed in well over a year but recently, every two days or so, the server just shuts down and cannot be accessed at all, I have to notify my service provider to physically restart the machine.

It seems to be tied to these SIGTERM errors I find in the logs. Problem is I have no idea how to fix these kinds of things or find the root cause as my skills in this area are lacking.

Anyone have any ideas what could be going on?

Apache/2.2.3 (CentOS)

20051115

Linux 2.6.18-028stab057.4 #1 SMP Fri Aug 1 10:47:59 MSD 2008 x86_64

From the error log: [Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down

Not entirely sure that's what's causing me my problems but it doesn't sound like a good thing.

I can post more log information if needed

[Sun Nov 01 05:07:17 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 05:07:17 2009] [notice] Digest: done
[Sun Nov 01 05:07:19 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 05:07:19 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 05:07:19 2009] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Sun Nov 01 05:07:19 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 15:16:11 2009] [error] [client 127.0.0.1] Invalid method in request get
[Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down
[Sun Nov 01 15:19:27 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 15:19:27 2009] [notice] Digest: done
[Sun Nov 01 15:19:27 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 16:42:31 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:59:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:37:42 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:14:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:40:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:10:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:35:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:46 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:25:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:26:03 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:56:57 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:21:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:16 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sun Nov 01 22:25:17 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:18 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:20 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:21 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:46:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:52:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:58 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:00:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:09:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:23:49 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:27:14 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:36:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:54:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:13:23 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:39:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:52:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:53:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:40 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:23:01 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:43:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:45:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:46:27 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:23:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:37 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:48 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:35:33 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 04:46:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 05:01:07 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 05:01:09 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 05:01:09 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:11 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 09:17:58 2009] [notice] caught SIGTERM, shutting down
[Mon Nov 02 09:18:58 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Digest: generating secret for digest authentication ...
[Mon Nov 02 09:18:58 2009] [notice] Digest: done
[Mon Nov 02 09:18:58 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Nov 02 09:32:34 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

Answer 1

from this page:

I found this info:

The mod_fastcgi process manager isn't particularly patient though (there's room for improvement here) and since it has to shutdown too, sends a SIGTERM to all of the FastCGI applications it is responsible for. Apache will restart the process manager and it will restart its managed applications (as if the server was just started). SIGTERM is, well, SIGTERM - your application should exit quickly.

What this implies to me is that if Database I/O, or some other part of the CGI script, fails to respond in a timely fashion (ie getting slower with data-volume growth), that mod_fastcgi is killing the script......is that how other people interpret these docs or what am I missing..

Answer 2

SIGTERM is used to restart Apache (provided that it's setup in init to auto-restart): http://httpd.apache.org/docs/2.2/stopping.html

The entries you see in the logs are almost certainly there because your provider used SIGTERM for that purpose. If it's truly crashing, not even serving static content, then that sounds like some sort of a thread/connection exhaustion issue. Perhaps a DoS that holds connections open?

Should definitely be something for your provider to investigate.

Answer 3

I had mysterious SIGTERM shutdowns in our L.A.M.P. server, and it turned out to be an error in a custom PHP module, which was caused by mismatched versions. It was found by looking in the apache access/error logs at the time of malfunction. Don't forget to turn error logging on.

Answer 4

Have you asked your provider to investigate? I assume this is not a dedicated server,

On the face of it, this seems like a security exception and somone is trying to exploit it / or there is a process running at a set time which is causing this, can you think of anything that runs on the server every 2 days? Logging tools?

SIGTERM is the signal sent to a process to request its termination. The symbolic constant for SIGTERM is defined in the header file signal.h. Symbolic signal names are used because signal numbers can vary across platforms, however on the vast majority of systems, SIGTERM is signal #15.