MIPS32 router: module_init not called for kernel m

Question

I'm developing a kernel module that I want to run on my router. The router model is DGN2200v2 by Netgear. It's running Linux 2.6.30 on MIPS. My problem is that when I load my module it seems that my module_init isn't getting called. I tried to narrow it down by modifying my module_init to return -3 (which indicates an error?) and insmod still reports success. I can see my module in the output of lsmod, but I don't see my printk output using dmesg.

For starters, I wanted to create the simplest possible module:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>

static int my_init(void)
{
    printk(KERN_EMERG "init_module() called\n");
    return -3;
}

static void my_cleanup(void)
{
    printk(KERN_EMERG "cleanup_module() called\n");
}

module_init(my_init);
module_exit(my_cleanup);

This is the Makefile I'm using:

TOOLCHAIN=/home/user/buildroot-2016.08/output/host/usr/bin/mips-buildroot-linux-uclibc-
ARCH=mips
CC = $(TOOLCHAIN)gcc

KBUILD_CFLAGS:=.

EXTRA_CFLAGS := -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/include\
  -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/arch/mips/include/asm/mach-mipssim\
  -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/arch/mips/include/asm/mach-generic\
  -fno-pic -mno-abicalls -O2

obj-m := module.o
KDIR := /home/user/buildroot-2016.08/output/build/linux-headers-2.6.30
PWD := $(shell pwd)

default:
    $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules

I'm running make like so:

make ARCH=mips CROSS_COMPILE=/home/user/buildroot-2016.08/output/host/usr/bin/mips-buildroot-linux-uclibc-

which passes successfully.

As you can see, I'm using Buildroot which I (hopefully) configured correctly. I can paste my .config if needed.

I ran objdump on my module and didn't find a problem. In particular, the module_init symbol seems to point to the same place as my my_init function, and it seems to have the code I expect it to:

module.ko:     file format elf32-tradbigmips
module.ko
architecture: mips:isa32, flags 0x00000011:
HAS_RELOC, HAS_SYMS
start address 0x00000000
private flags = 50001001: [abi=O32] [mips32] [not 32bitmode] [noreorder]

MIPS ABI Flags Version: 0

ISA: MIPS32
GPR size: 32
CPR1 size: 0
CPR2 size: 0
FP ABI: Soft float
ISA Extension: None
ASEs:
    None
FLAGS 1: 00000001
FLAGS 2: 00000000

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .MIPS.abiflags 00000018  00000000  00000000  00000038  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE
  1 .reginfo      00000018  00000000  00000000  00000050  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE
  2 .note.gnu.build-id 00000024  00000018  00000018  00000068  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .text         00000040  00000000  00000000  00000090  2**4
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  4 .rodata.str1.4 00000038  00000000  00000000  000000d0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .modinfo      0000005c  00000000  00000000  00000108  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .data         00000000  00000000  00000000  00000170  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  7 .gnu.linkonce.this_module 0000014c  00000000  00000000  00000170  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA, LINK_ONCE_DISCARD
  8 .bss          00000000  00000000  00000000  000002c0  2**4
                  ALLOC
  9 .comment      00000040  00000000  00000000  000002c0  2**0
                  CONTENTS, READONLY
 10 .pdr          00000040  00000000  00000000  00000300  2**2
                  CONTENTS, RELOC, READONLY
 11 .gnu.attributes 00000010  00000000  00000000  00000340  2**0
                  CONTENTS, READONLY
 12 .mdebug.abi32 00000000  00000000  00000000  00000350  2**0
                  CONTENTS, READONLY
SYMBOL TABLE:
00000000 l    d  .MIPS.abiflags 00000000 .MIPS.abiflags
00000000 l    d  .reginfo   00000000 .reginfo
00000018 l    d  .note.gnu.build-id 00000000 .note.gnu.build-id
00000000 l    d  .text  00000000 .text
00000000 l    d  .rodata.str1.4 00000000 .rodata.str1.4
00000000 l    d  .modinfo   00000000 .modinfo
00000000 l    d  .data  00000000 .data
00000000 l    d  .gnu.linkonce.this_module  00000000 .gnu.linkonce.this_module
00000000 l    d  .bss   00000000 .bss
00000000 l    d  .comment   00000000 .comment
00000000 l    d  .pdr   00000000 .pdr
00000000 l    d  .gnu.attributes    00000000 .gnu.attributes
00000000 l    d  .mdebug.abi32  00000000 .mdebug.abi32
00000000 l    df *ABS*  00000000 module.c
00000000 l     F .text  0000002c my_init
0000002c l     F .text  00000014 my_cleanup
00000000 l       .rodata.str1.4 00000000 $LC0
0000001c l       .rodata.str1.4 00000000 $LC1
00000000 l    df *ABS*  00000000 module.mod.c
00000000 l     O .modinfo   00000023 __mod_srcversion23
00000024 l     O .modinfo   00000009 __module_depends
00000030 l     O .modinfo   0000002c __mod_vermagic5
00000000 g     O .gnu.linkonce.this_module  0000014c __this_module
0000002c g     F .text  00000014 cleanup_module
00000000 g     F .text  0000002c init_module
00000000         *UND*  00000000 printk



Disassembly of section .MIPS.abiflags:

00000000 <.MIPS.abiflags>:
   0:   00002001    movf    a0,zero,$fcc0
   4:   01000003    0x1000003
    ...
  10:   00000001    movf    zero,zero,$fcc0
  14:   00000000    nop

Disassembly of section .reginfo:

00000000 <.reginfo>:
   0:   a2000014    sb  zero,20(s0)
    ...
  14:   00007fef    0x7fef

Disassembly of section .note.gnu.build-id:

00000018 <.note.gnu.build-id>:
  18:   00000004    sllv    zero,zero,zero
  1c:   00000014    0x14
  20:   00000003    sra zero,zero,0x0
  24:   474e5500    c1  0x14e5500
  28:   c8e5d654    lwc2    $5,-10668(a3)
  2c:   cb477d3d    lwc2    $7,32061(k0)
  30:   dfa48d71    ldc3    $4,-29327(sp)
  34:   c2ea16da    ll  t2,5850(s7)
  38:   f6bcae7d    sdc1    $f28,-20867(s5)

Disassembly of section .text:

00000000 <init_module>:
   0:   27bdffe8    addiu   sp,sp,-24
   4:   3c040000    lui a0,0x0
            4: R_MIPS_HI16  $LC0
   8:   3c020000    lui v0,0x0
            8: R_MIPS_HI16  printk
   c:   afbf0014    sw  ra,20(sp)
  10:   24420000    addiu   v0,v0,0
            10: R_MIPS_LO16 printk
  14:   0040f809    jalr    v0
  18:   24840000    addiu   a0,a0,0
            18: R_MIPS_LO16 $LC0
  1c:   8fbf0014    lw  ra,20(sp)
  20:   2402fffd    li  v0,-3
  24:   03e00008    jr  ra
  28:   27bd0018    addiu   sp,sp,24

modinfo output also matches what I expect (same modinfo output as for another .ko that's found on the router, except for the srcversion which my module has but the other module on the router doesn't):

filename:       /home/user/module/module.ko
srcversion:     B0BADBA395A121CF49B74DC
depends:        
vermagic:       2.6.30 mod_unload MIPS32_R1 32BIT 

It's entirely possible that I messed something up in my Buildroot configuration, or something doesn't quite match the CPU type of the router, but my init code is so minimal that I'm out of ideas as to what could be wrong.

Answer 1

It turns out that the problem was related to a different kernel configuration between my development environment and the router. Specifically, my kernel was using CONFIG_UNUSED_SYMBOLS whereas the router's was not.

The reason this caused a problem even in a trivial module is that when the kernel loads a module it doesn't only look up the module_init symbol in the module's symbol table. Rather, it reads the module struct from the module (from the .gnu.linkonce.this_module section), and then calls the init module through that struct.

The offset of the init function pointer inside the module struct depends on the kernel configuration, which explains why the kernel can't find the init function if the configuration is different.

Thanks to Sam Protsenko for investing a lot of time in helping me crack this!