addr2line on kernel module

Question

I'm trying to debug kernel module. I suspect to have there some memory leaks. To check it I have prepared build with enabled Memory leak debugging for kernel and modules. And I got some warning from that:

[11839.429168] slab error in verify_redzone_free(): cache `size-64': memory outside object was overwritten
[11839.438659] [<c005575c>] (unwind_backtrace+0x0/0x164) from [<c0116ca0>] (kfree+0x278/0x4d8)
[11839.447357] [<c0116ca0>] (kfree+0x278/0x4d8) from [<bf083f48>] (some_function+0x18/0x1c [my_module])
[11839.457214] [<bf083f48>] (some_function+0x18/0x1c [my_module]) from [<bf08762c>] (some_function+0x174/0x718 [my_module])
[11839.470184] [<bf08762c>] (some_function+0x174/0x718 [my_module]) from [<bf0a56b8>] (some_function+0x12c/0x16c [my_module])
[11839.483917] [<bf0a56b8>] (some_function+0x12c/0x16c [my_module]) from [<bf085790>] (some_function+0x8/0x10 [my_module])
[11839.496368] [<bf085790>] (some_function+0x8/0x10 [my_module]) from [<bf07b74c>] (some_function+0x358/0x6d4 [my_module])
[11839.507476] [<bf07b74c>] (some_function+0x358/0x6d4 [my_module]) from [<c00a60f8>] (worker_thread+0x1e8/0x284)
[11839.517211] [<c00a60f8>] (worker_thread+0x1e8/0x284) from [<c00a9edc>] (kthread+0x78/0x80)
[11839.525543] [<c00a9edc>] (kthread+0x78/0x80) from [<c004f8fc>] (kernel_thread_exit+0x0/0x8)

There is no problem to translate addresses which points to kernel:

$ addr2line -f -e vmlinux.kmeml c0116ca0
verify_redzone_free
/[...]/kernel/mm/slab.c:2922

But I can't do that if addresses are from my_module:

$ addr2line -f -e vmlinux.kmeml bf0a56b8
??
??:0

I was also trying with module file:

$ addr2line -f -e my_module.ko bf0a56b8
??
??:0

How can I translate this addresses to files and line numbers?

Answer 1

Maybe you should use -g parameter to compile the module.

Answer 2

I suppose the module is built with debug info included. If so, you can use gdb or objdump to find out which source file and line each address belongs to. Something like this:

$ gdb "$(modinfo -n my_module)"
(gdb) list *(some_function+0x12c)

Gdb will now tell the name of the source file and the line in it.

You can also do a similar thing with objdump but it is a bit more difficult. First, disassemble the module:

objdump -dSlr my_module.ko > my_module.disasm

When called with -S option, objdump will include the source lines in the resulting listing where appropriate.

You can now scroll the listing down to the code of some_function, find the instruction at offset 0x12c from the beginning of the function. The source line will be indicated above it.

EDIT:

After many experiments, I found that although addr2line can indeed be used for kernel modules, eu-addr2line (a similar tool from elfutils) seems to be more reliable. That is, sometimes addr2line output incorrect source lines but eu-add2line did things right.

To use eu-addr2line, one may need to install libdw and libebl libraries if they are not already installed along with elfutils.

The usage is similar to that of addr2line:

eu-addr2line -f -e <path_to_the_module> -j <section_name> <offset_in_section>

If the debug information for a kernel module is stored in separate file (this is often the case for the kernels provided by the major Linux distros), the path to that file should be used as <path_to_the_module>.

Answer 3

You indeed need to run addr2line on your kernel module and not kernel but there is a twist -

the kernel module file uses relative addresses, the crash address you have is actually composed of:

offset inside module + module load address is memory.

So what you need to do is find the kernel moduel load address is memory first by doing cat /proc/modules, finding to what module that address belongs to, in case you don't know, subtract the module load address from the crash address and feed that to addr2line

good luck