I am running a command-line PHP job that is resulting in a Segmentation Fault. This job has worked for a long time, but it processes something that is emailed in. Apparently there's something in this email that's break it, but I have no idea what. If I put the core file in gdb, it's really no help:
$ gdb /usr/local/bin/php core.20381
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/php...done.
[New Thread 20381]
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libmysqlclient_r.so.15...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmysqlclient_r.so.15
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libmcrypt.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpng12.so.0
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl-gnutls.so.4
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Can't read symbols from system-supplied DSO at 0x7fffb4bfc000: File truncated
Core was generated by `/usr/local/bin/php /src/prod/current/utils/cron.php --run=/utils/proces'.
Program terminated with signal 11, Segmentation fault.
#0 match (
eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3aa "N",
mstart=0x13708600 "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1\">\n\n</head>\n\n<body style=\"padding"..., markptr=0x0, offset_top=4, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470
470 {
If I do a 'bt', I just get this "match" statement over and over again. Here are the first few:
#1 0x000000000046ef1e in match (
eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,
offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17679) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#2 0x0000000000479789 in match (
eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>,
markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#3 0x000000000046ef1e in match (
eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,
offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17677) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#4 0x0000000000479789 in match (
eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>,
markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#5 0x000000000046ef1e in match (
eptr=0x1370a885 "\t</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,
offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17675) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
I'm really at a loss here. I've tried running the command in strace, and it seems to be seg-faulting in the PEAR Mail_Queue library. Here's the end of the strace (some identifying information has been removed):
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
open("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
mmap(NULL, 55723, PROT_READ, MAP_SHARED, 6, 0) = 0x2b49f7bb4000
brk(0x987b000) = 0x987b000
brk(0x98bb000) = 0x98bb000
munmap(0x2b49f7bb4000, 55723) = 0
close(6) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
fcntl(6, F_SETFL, O_RDONLY) = 0
fcntl(6, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(6, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(6, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("IP ADDRESS TO DB SERVER REMOVED")}, 16) = -1 EINPROGRESS (Operation now in progress)
fcntl(6, F_SETFL, O_RDWR) = 0
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
setsockopt(6, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(6, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
poll([{fd=6, events=POLLIN}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
read(6, "C\0\0\0\n5.1.37-1ubuntu5.5-log\0Q\257\3\0009"..., 16384) = 71
write(6, "=\0\0\1\205\242\2\0\0\0\0@\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65) = 65
read(6, "\7\0\0\2\0\0\0\2\0\0\0", 16384) = 11
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(6, "\10\0\0\0\2xxxxx", 12) = 12
read(6, "\7\0\0\1\0\0\0\2\0\0\0", 16384) = 11
lstat("/src/prod/current/www/application/pear/Mail/mime.php", {st_mode=S_IFREG|0770, st_size=50252, ...}) = 0
lstat("/src/prod/current/www/application/pear/Mail", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application", {st_mode=S_IFDIR|0776, st_size=4096, ...}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Segmentation fault
If anyone has any ideas or pointers to lines of investigation, I'd truly appreciate it. I'm really stumped on this one.