Python server “Aborted (Core dumped)”

Question

I use web.py to create a Python web server. This server is called to solve linear programming problems, and it uses the library CBC to do that.

Every once in a while, the server crashes with a log that looks like that:

78.243.184.3:56271 - - [03/Jun/2016 04:35:54] "HTTP/1.1 GET /optimization" - 200 OK
Aborted (core dumped)

I belive "Aborted (core dumped)" is a C error, so it comes from either web.py or CBC.

Is there any way to trace back the source of the error?

Answer 1

A core dump is caused by a fault in the native code in your web server. Python is pretty darn solid these days, so such faults are almost always caused by bugs in C extensions in my experience.

You therefore have 3 problems.

1. You need to find the core dump file. This is usually in the current working directory of the process that dumped. However, there are configation options that can change this.

2. You need to debug the call stack of what has failed. That is covered in StackOverflow - see How to analyze a program's core dump file with gdb?

3. You might need to relate the Python interpreter stack back to the Python code that you were running. To do this you will need to install the Python debug symbols and Python extensions for gdb. The Python wiki has good advice on how to do that here.

Answer 2

Most common reason for a Core dump is to access a memory address outside of your reach (memory that do not belong to your program). The operatingsystem interrupts the program with a interruption called SegFault or BusError depeding on how the program tried to access the invalid memory address. The program will then be forceful closed down by the kernel. If the kernel has been configured to create a core-dump (a dump of memory and stack of the program) one will be saved to disc. As noted in the other answer you may load the core-dump into GDB or other debugger and display what the program was doing at the time it crashed. This might or might not give you the problem at hand. Normally it's hard even for a experienced programmer to used these tools so be aware. If you want to try to use GDB, try this:

$ gdb /path/to/crashing/program/binary /path/to/core

(gdb) bt

'bt' will display 'backtraces' otherwise known as StackTraces, and can be useful for a programmer to track down the error.

If you are able to reproduce the error, you are probably lucker submitting a detailed bugreport to the creator of the program in question. Even me as a senior software developer takes that route from time to time. :-)