The version of GDB that is available on HPUX has a command called "packcore", which creates a tarball containing the core dump, the executable and all libraries. I've found this extremely useful when trying to debug core dumps on a different machine.
Is there a similar command in the standard version of GDB that I might find on a Linux machine?
I'm looking for an easy command that someone that isn't necessarily a developer can run when things go bad on a production machine.
The core file includes the command from which it was generated. Ideally this will include the full path to the appropriate executable. For example:
$ file core.29529
core.29529: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/bin/sleep 60'
Running ldd
on an ELF binary will show what libraries it depends on:
$ ldd /bin/sleep
linux-vdso.so.1 => (0x00007fff1d3ff000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d3ce00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d3ca00000)
So now I know the executable and the libraries needed to analyze the core dump.
The tricky part here is extracting the executable path from the core file. There doesn't appear to be a good tool for reading this directly. The data is encoded in a prpsinfo structure (from /usr/include/sys/procfs.h
), and you can find the location size of the data using readelf
:
$ readelf -n core.29529
Notes at offset 0x00000468 with length 0x00000558:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
CORE 0x00000088 NT_PRPSINFO (prpsinfo structure)
CORE 0x00000130 NT_AUXV (auxiliary vector)
CORE 0x00000200 NT_FPREGSET (floating point registers)
...so one could in theory write a code snippet to extract the command line from this structure and print it out in a way that would make this whole process easier to automate. You could, of course, just parse the output of file
:
$ file core.29529 | sed "s/.*from '\([^']*\)'/\1/"
/bin/sleep 60
So that's all the parts. Here's a starting point for putting it all together:
#!/bin/sh
core=$1
exe=$(file $core | sed "s/.*from '\([^']*\)'/\1/" | awk '{print $1}')
libs=$(
ldd $exe |
awk '
/=> \// {print $3}
! /=>/ {print $1}
'
)
cat <<EOF | tar -cah -T- -f $1-all.tar.xz
$libs
$exe
EOF
For my example, if I name this script packcore
and run it on the core file from the sleep
command, I get this:
$ packcore core.29529
tar: Removing leading `/' from member names
$ tar -c -f core.29529-all.tar.xz
core.29529
lib64/libc.so.6
lib64/ld-linux-x86-64.so.2
bin/sleep
As it stands this script is pretty fragile; I've made lots of assumptions about the output from ldd
based on only this sample output.
Here's a script that does the necessary steps (tested only on RHEL5, but might work elsewhere too):
#!/bin/sh
#
# Take a core dump and create a tarball of all of the binaries and libraries
# that are needed to debug it.
#
include_core=1
keep_workdir=0
usage()
{
argv0="$1"
retval="$2"
errmsg="$3"
if [ ! -z "$errmsg" ] ; then
echo "ERROR: $errmsg" 1>&2
fi
cat <<EOF
Usage: $argv0 [-k] [-x] <corefile>
Parse a core dump and create a tarball with all binaries and libraries
needed to be able to debug the core dump.
Creates <corefile>.tgz
-k - Keep temporary working directory
-x - Exclude the core dump from the generated tarball
EOF
exit $retval
}
while [ $# -gt 0 ] ; do
case "$1" in
-k)
keep_workdir=1
;;
-x)
include_core=0
;;
-h|--help)
usage "$0" 0
;;
-*)
usage "$0" 1 "Unknown command line arguments: $*"
;;
*)
break
;;
esac
shift
done
COREFILE="$1"
if [ ! -e "$COREFILE" ] ; then
usage "$0" 1 "core dump '$COREFILE' doesn't exist."
fi
case "$(file "$COREFILE")" in
*"core file"*)
break
;;
*)
usage "$0" 1 "per the 'file' command, core dump '$COREFILE' is not a core dump."
;;
esac
cmdname=$(file "$COREFILE" | sed -e"s/.*from '\(.*\)'/\1/")
echo "Command name from core file: $cmdname"
fullpath=$(which "$cmdname")
if [ ! -x "$fullpath" ] ; then
usage "$0" 1 "unable to find command '$cmdname'"
fi
echo "Full path to executable: $fullpath"
mkdir "${COREFILE}.pack"
gdb --eval-command="quit" "${fullpath}" ${COREFILE} 2>&1 | \
grep "Reading symbols" | \
sed -e's/Reading symbols from //' -e's/\.\.\..*//' | \
tar --files-from=- -cf - | (cd "${COREFILE}.pack" && tar xf -)
if [ $include_core -eq 1 ] ; then
cp "${COREFILE}" "${COREFILE}.pack"
fi
tar czf "${COREFILE}.pack.tgz" "${COREFILE}.pack"
if [ $keep_workdir -eq 0 ] ; then
rm -r "${COREFILE}.pack"
fi
echo "Done, created ${COREFILE}.path.tgz"