stackwalker loads all symbol files, but still does

Question

I'm probably doing something wrong, but I can't figure this one out.

I have a simple crash minidump, generated on Windows. If I open the dump in visual studio, it loads without problems and show the crash line. But I cannot get it to symbolicate in minidump_stackwalker.

I did create a symbol repository folder, inside I have: symbols/myapplication.pdb/67892B042C8F4CCFAA6BE53445F9B2141/myapplication.sym and all the: symbols/wkernel32.pdb/XXXXXX/wkernel32.sym etc used by the application.

When I call "minidump_stackwalk mycrash.dmp symbols", stderr shows that simple_symbol_supplier finds all the symbols correctly, and load them. However, the output keeps being the stack without any symbols in it.

I also tried to compile and run minidump_stackwalk on linux and on MacOSX, but the result is the same.

What am I doing wrong?

The stderr output of stackwalk looks like this:

2014-05-08 16:44:48: minidump_processor.cc:235: INFO: Processing minidump in file 604c29d0-318a-4321-9e40-b0198085c17d.dmp
2014-05-08 16:44:48: minidump.cc:3258: INFO: Minidump opened minidump 604c29d0-318a-4321-9e40-b0198085c17d.dmp on fd 3
2014-05-08 16:44:48: minidump.cc:3303: INFO: Minidump not byte-swapping minidump
2014-05-08 16:44:48: minidump.cc:1726: ERROR: MinidumpModule could not determine debug_file for C:\Windows\winsxs\x86_microsoft.vc90.mfcloc_1fc8b3b9a1e18e3b_9.0.30729.6161_none_49768ef57548175e\MFC90ENU.DLL
2014-05-08 16:44:48: minidump.cc:1794: ERROR: MinidumpModule could not determine debug_identifier for C:\Windows\winsxs\x86_microsoft.vc90.mfcloc_1fc8b3b9a1e18e3b_9.0.30729.6161_none_49768ef57548175e\MFC90ENU.DLL
2014-05-08 16:44:48: minidump_processor.cc:103: INFO: Minidump 604c29d0-318a-4321-9e40-b0198085c17d.dmp has CPU info, OS info, Breakpad info, exception, module list, thread list, dump thread, and requesting thread
2014-05-08 16:44:48: minidump_processor.cc:137: INFO: Looking at thread 604c29d0-318a-4321-9e40-b0198085c17d.dmp:0/24 id 0x1570
2014-05-08 16:44:49: basic_source_line_resolver.cc:223: INFO: Loading symbols for module C:\Program Files\MyApplication\myapplication.exe from buffer
2014-05-08 16:44:49: basic_source_line_resolver.cc:223: INFO: Loading symbols for module C:\Windows\System32\user32.dll from buffer
2014-05-08 16:44:49: basic_source_line_resolver.cc:223: INFO: Loading symbols for module C:\Windows\System32\kernel32.dll from buffer
2014-05-08 16:44:49: basic_source_line_resolver.cc:223: INFO: Loading symbols for module C:\Windows\System32\ntdll.dll from buffer
2014-05-08 16:44:49: minidump_processor.cc:137: INFO: Looking at thread 604c29d0-318a-4321-9e40-b0198085c17d.dmp:1/24 id 0xbb4
...
2014-05-08 16:44:49: minidump_processor.cc:229: INFO: Processed 604c29d0-318a-4321-9e40-b0198085c17d.dmp
2014-05-08 16:44:49: minidump.cc:3232: INFO: Minidump closing minidump on fd 3

And the output looks like this:

Operating system: Windows NT
                  6.1.7601 Service Pack 1
CPU: x86
     GenuineIntel family 6 model 15 stepping 6
     2 CPUs

Crash reason:  EXCEPTION_ACCESS_VIOLATION
Crash address: 0x0

Thread 0 (crashed)
 0  myapplication.exe + 0x6d0aa7
    eip = 0x01500aa7   esp = 0x0023eef4   ebp = 0x0023eefc   ebx = 0x0505f9bc
    esi = 0x0505f9bc   edi = 0x00000000   eax = 0x0514fe33   ecx = 0x0003c11d
    edx = 0x00000003   efl = 0x00010202
    Found by: given as instruction pointer in context
 1  myapplication.exe + 0x146fb
    eip = 0x00e446fc   esp = 0x0023ef04   ebp = 0x0023ef20
    Found by: previous frame's frame pointer
 2  myapplication.exe + 0x127b1
    eip = 0x00e427b2   esp = 0x0023ef28   ebp = 0x0023ef4c
    Found by: previous frame's frame pointer
...

Answer 1

Some extra informations:

http://www.chromium.org/developers/decoding-crash-dumps

http://blog.inventic.eu/2012/08/qt-and-google-breakpad/

My core problem was that in symbols folder i don't have folder with exactly same name as my binary. Simply everything name as your binary. Example on linux:

$google-breakpad/src/tools/linux/dump_syms/dump_syms ./MyBinary > MyBinary.sym
$head -n1 MyBinary.sym

it will print something like

MODULE Linux x86_64 6EDC6ACDB282125843FD59DA9C81BD830 MyBinary

Now make something like:

$ mkdir -p ./symbols/MyBinary/6EDC6ACDB282125843FD59DA9C81BD830
$ mv test.sym ./symbols/MyBinary/6EDC6ACDB282125843FD59DA9C81BD830

And on the end

google-breakpad/src/processor/minidump_stackwalk 09fd98ec-d55c-29e1-4ae067b0-4aaec0d6.dmp ./symbols

Or you can use simple script:

Example usage:

./script.py --dump_syms=/home/user/projects/libs/google-breakpad/src/tools/linux/dump_syms/dump_syms \
--minidump_stackwalk=/home/user/projects/libs/google-breakpad/src/processor/minidump_stackwalk \
--binary=./MyBinary \
--output_dir=/home/user/.config/MyBinary/dumps/
--dump_file=/home/user/.config/MyBinary/dumps/75ad100c-ef7c-c4eb-1da750f8-45586148.dmp

script:

#!/usr/bin/python

import sys
import getopt
import os
import errno
from os.path import basename

def make_sure_path_exists(path):
    try:
        os.makedirs(path)
    except OSError as exception:
        if exception.errno != errno.EEXIST:
            raise


def main(argv):

    dumpSymsExe = ""
    binary = ""
    outputDir = ""
    minidumpExe = ""
    dumpFile = ""

    try:                                
        opts, args = getopt.getopt(argv, "", ["help", "dump_syms=","binary=",
                                              "output_dir=","dump_file=",
                                              "minidump_stackwalk="])
    except getopt.GetoptError:
        print ("Please pass correct arguments!")
        sys.exit(2)  

    for opt, arg in opts:
        if opt == "--dump_syms":
            dumpSymsExe = arg
        if opt == "--binary":
            binary = arg
        if opt == "--output_dir":
            outputDir = arg
        if opt == "--dump_file":
            dumpFile = arg
        if opt == "--minidump_stackwalk":
            minidumpExe = arg

    outputFile =  outputDir + basename(binary) + ".sym"
    command = '{0} {1} > {2}'.format(dumpSymsExe,binary,outputFile)

    print ('Running {0}'.format(command) )
    os.system(command)

    symFile = open(outputFile, 'r')
    firstLine = symFile.readline()

    print ('First line {0}'.format(firstLine))
    lineArguments = firstLine.split(' ')
    print ('Magic file {0}'.format(lineArguments[-2]))

    symbolsDir = (outputDir + "symbols/" + basename(binary) + "/" +
                  lineArguments[-2] + "/")

    make_sure_path_exists(symbolsDir)
    os.rename(outputFile,symbolsDir + basename(outputFile))

    command = '{0} {1} {2}'.format(minidumpExe,dumpFile,outputDir + "symbols")
    print command
    os.system(command)

if __name__ == "__main__":
    main(sys.argv[1:])

---

Another thing can be if you look at end of your detailed crash info you will see something like:

0x7f75e2695000 - 0x7f75e28aefff  librtmp.so.0  ???
0x7f75e28af000 - 0x7f75e2ae1fff  libidn.so.11.6.11  ???
0x7f75e2ae2000 - 0x7f75e2d0bfff  libexpat.so.1.6.0  ???
0x7f75e2d0c000 - 0x7f75e2fadfff  libfreetype.so.6.10.1  ???
0x7f75e2fae000 - 0x7f75e31b7fff  libXrandr.so.2.2.0  ???
0x7f75e31b8000 - 0x7f75e33bbfff  libdl-2.17.so  ???
0x7f75e33bc000 - 0x7f75e35c7fff  libdrm.so.2.4.0  ???
0x7f75e35c8000 - 0x7f75e37cdfff  libXxf86vm.so.1.0.0  ???
0x7f75e37ce000 - 0x7f75e39ebfff  libxcb.so.1.1.0  ???
0x7f75e39ec000 - 0x7f75e3bf0fff  libxcb-dri2.so.0.0.0  ???
0x7f75e3bf1000 - 0x7f75e3e07fff  libxcb-glx.so.0.0.0  ???
0x7f75e3e08000 - 0x7f75e4009fff  libX11-xcb.so.1.0.0  ???
0x7f75e400a000 - 0x7f75e420ffff  libXfixes.so.3.1.0  ???
0x7f75e4210000 - 0x7f75e4412fff  libXdamage.so.1.1.0  ???
0x7f75e4413000 - 0x7f75e4624fff  libXext.so.6.4.0  ???
0x7f75e4625000 - 0x7f75e4849fff  libglapi.so.0.0.0  ???
0x7f75e484b000 - 0x7f75e4aa9fff  libcurl-gnutls.so.4.3.0  ???  (WARNING: No symbols, libcurl-gnutls.so.4.3.0, B662FA5595A461FACCB60A9D323597830)
0x7f75e4aaa000 - 0x7f75e4d26fff  libGLEW.so.1.8.0  ???
0x7f75e4d2b000 - 0x7f75e505ffff  libX11.so.6.3.0  ???
0x7f75e5060000 - 0x7f75e5278fff  libz.so.1.2.8  ???
0x7f75e5279000 - 0x7f75e54b4fff  libfontconfig.so.1.7.0  ???
0x7f75e54b5000 - 0x7f75e56c5fff  libglfw.so.2.6  ???
0x7f75e56c6000 - 0x7f75e5a88fff  libc-2.17.so  ???  (WARNING: No symbols, libc-2.17.so, 8A0DCFA21992EC412D98B231AF91DC1D0)
0x7f75e5a8e000 - 0x7f75e5ca3fff  libgcc_s.so.1  ???
0x7f75e5ca4000 - 0x7f75e5fa7fff  libm-2.17.so  ???
0x7f75e5fa8000 - 0x7f75e6296fff  libstdc++.so.6.0.18  ???  (WARNING: No symbols, libstdc++.so.6.0.18, 676136699F3BB3E58C19B83D8A296B660)
0x7f75e62ac000 - 0x7f75e64c4fff  libpthread-2.17.so  ???  (WARNING: No symbols, libpthread-2.17.so, 4CDAF5B4170C674F1A96E33F25274E0A0)
0x7f75e64c9000 - 0x7f75e6725fff  libGL.so.1.2.0  ???
0x7f75e6727000 - 0x7f75e69a4fff  libextension.so  ???  (WARNING: No symbols, libextension.so, 13E1DBC274C6099DEEF37259177DD16D0)
0x7f75e69a5000 - 0x7f75e6fa8fff  libcocos2d.so  ???
0x7f75e6fad000 - 0x7f75e6fcffff  ld-2.17.so  ???
0x7f75e6fde000 - 0x7f75e701ffff  libJsonCpp.so  ???
0x7f75e7021000 - 0x7f75e710efff  libEasySqlite.so  ???  (WARNING: No symbols, libEasySqlite.so, B4426B9A03AB419B458607C780DCCB1B0)
0x7f75e7142000 - 0x7f75e71ccfff  **libJansson.so**  ???  (**WARNING: No symbols**, libJansson.so, 31CEB5F4D63244C3BF657C501398D8650)

As you can see i don't have symbols for libJansson.so and many others that means that in my "symbols" folder i need to have symbols for this library.

I can generate them in similar way

$google-breakpad/src/tools/linux/dump_syms/dump_syms ./libJansson.so > libJansson.so.sym
$head -n1 libJansson.so.sym

it will print something like

MODULE Linux x86_64 65C874A94F8B264EAE501D2694DFCBA60 libJansson.so

Now make something like:

$ mkdir -p ./symbols/libJansson.so/65C874A94F8B264EAE501D2694DFCBA60
$ mv test.sym ./symbols/libJansson.so/65C874A94F8B264EAE501D2694DFCBA60

and now i can see stack trace for libJansson.so

So if you use my script you must only do something like this:

./script.py --dump_syms=/home/user/projects/libs/google-breakpad/src/tools/linux/dump_syms/dump_syms \
--minidump_stackwalk=/home/user/projects/libs/google-breakpad/src/processor/minidump_stackwalk \
--binary=./MyBinary --output_dir=/home/user/.config/MyBinary/dumps/ \
--dump_file=/home/user/.config/MyBinary/dumps/75ad100c-ef7c-c4eb-1da750f8-45586148.dmp

./script.py --dump_syms=/home/user/projects/libs/google-breakpad/src/tools/linux/dump_syms/dump_syms \
--minidump_stackwalk=/home/user/projects/libs/google-breakpad/src/processor/minidump_stackwalk \
--binary=./libJansson.so --output_dir=/home/user/.config/MyBinary/dumps/ \
--dump_file=/home/user/.config/MyBinary/dumps/75ad100c-ef7c-c4eb-1da750f8-45586148.dmp

Answer 2

use the "minidump_stackwalk" in linux to parse the '.sym' file generated on windows. But,firstly, you must transform the '.sym' from dos format to unix format and the encoding type to UTF-8. if you skip it,the problem you asked above would appear.