LD_PRELOAD and linkage

Question

I have this small testcode atfork_demo.c:

#include <stdio.h>
#include <pthread.h>

void hello_from_fork_prepare() {
    printf("Hello from atfork prepare.\n");
    fflush(stdout);
}

void register_hello_from_fork_prepare() {
    pthread_atfork(&hello_from_fork_prepare, 0, 0);
}

Now, I compile it in two different ways:

gcc -shared -fPIC atfork_demo.c -o atfork_demo1.so
gcc -shared -fPIC atfork_demo.c -o atfork_demo2.so -lpthread

My demo main atfork_demo_main.c is this:

#include <dlfcn.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, const char** argv) {
    if(argc <= 1) {
        printf("usage: ... lib.so\n");
        return 1;
    }
    void* plib = dlopen("libpthread.so.0", RTLD_NOW|RTLD_GLOBAL);
    if(!plib) {
        printf("cannot load pthread, error %s\n", dlerror());
        return 1;
    }
    void* lib = dlopen(argv[1], RTLD_LAZY);
    if(!lib) {
        printf("cannot load %s, error %s\n", argv[1], dlerror());
        return 1;
    }
    void (*reg)();
    reg = dlsym(lib, "register_hello_from_fork_prepare");
    if(!reg) {
        printf("did not found func, error %s\n", dlerror());
        return 1;
    }
    reg();
    fork();
}

Which I compile like this:

gcc atfork_demo_main.c -o atfork_demo_main.exec -ldl

Now, I have another small demo atfork_patch.c where I want to override pthread_atfork:

#include <stdio.h>
int pthread_atfork(void (*prepare)(void), void (*parent)(void), void (*child)(void)) {
  printf("Ignoring pthread_atfork call!\n");
  fflush(stdout);
  return 0;
}

Which I compile like this:

gcc -shared -O2 -fPIC patch_atfork.c -o patch_atfork.so

And then I set LD_PRELOAD=./atfork_patch.so, and do these two calls:

./atfork_demo_main.exec ./atfork_demo1.so
./atfork_demo_main.exec ./atfork_demo2.so

In the first case, the LD_PRELOAD-override of pthread_atfork worked and in the second, it did not. I get the output:

Ignoring pthread_atfork call!
Hello from atfork prepare.

So, now to the question(s):

• Why did it not work in the second case?
• How can I make it work also in the second case, i.e. also override it? In my real use case, atfork_demo is some library which I cannot change. I also cannot change atfork_demo_main but I can make it load any other code. I would prefer if I can just do it with some change in atfork_patch.

You get some more debug output if you also use LD_DEBUG=all. Maybe interesting is this bit, for the second case:

   841:     symbol=__register_atfork;  lookup in file=./atfork_demo_main.exec [0]
   841:     symbol=__register_atfork;  lookup in file=./atfork_patch_extended.so [0]
   841:     symbol=__register_atfork;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
   841:     symbol=__register_atfork;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
   841:     binding file ./atfork_demo2.so [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__register_atfork' [GLIBC_2.3.2]

So, it searches for the symbol __register_atfork. I added that to atfork_patch_extended.so but it doesn't find it and uses it from libc instead. How can I make it find and use my __register_atfork?

---

As a side note, my main goal is to ignore the atfork handlers when fork() is called, but this is not the question here, but actually here. One solution to that, which seems to work, is to override fork() itself by this:

pid_t fork(void) {
  return syscall(SYS_clone, SIGCHLD, 0);
}

Answer 1

It doesn't work for the second case because there is nothing to override. Your second library is linked statically with pthread library:

$ readelf --symbols atfork_demo1.so | grep pthread_atfork
     7: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND pthread_atfork
    54: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND pthread_atfork
$ readelf --symbols atfork_demo2.so | grep pthread_atfork
    41: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS pthread_atfork.c
    47: 0000000000000830    31 FUNC    LOCAL  DEFAULT   12 __pthread_atfork
    49: 0000000000000830    31 FUNC    LOCAL  DEFAULT   12 pthread_atfork

So it will use local pthread_atfork each time, regardless of LD_PRELOAD or any other loaded libraries.

How to overcome that? Looks like for described configuration it is not possible since you need to modify atfork_demo library or main executable anyway.

Answer 2

Before answering this question, I would stress that this is a really bad idea for any production application.

If you are using a third party library that puts such constraints in place, then think about an alternative solution, such as forking early to maintain a "helper" process, with a pipe between you and it... then, when you need to call exec(), you can request that it does the work (fork(), exec()) on your behalf.

Patching or otherwise side-stepping the services of a system call such as pthread_atfork() is just asking for trouble (missed events, memory leaks, crashes, etc...).

---

As @Sergio pointed out, pthread_atfork() is actually built into atfork_demo2.so, so you can't do anything to override it... However examining the disassembly / source of pthread_atfork() gives you a decent hint about how achieve what you're asking:

0000000000000830 <__pthread_atfork>:
 830:   48 8d 05 f9 07 20 00    lea    0x2007f9(%rip),%rax        # 201030 <__dso_handle>
 837:   48 85 c0                test   %rax,%rax
 83a:   74 0c                   je     848 <__pthread_atfork+0x18>
 83c:   48 8b 08                mov    (%rax),%rcx
 83f:   e9 6c fe ff ff          jmpq   6b0 <__register_atfork@plt>
 844:   0f 1f 40 00             nopl   0x0(%rax)
 848:   31 c9                   xor    %ecx,%ecx
 84a:   e9 61 fe ff ff          jmpq   6b0 <__register_atfork@plt>

or the source (from here):

int
pthread_atfork (void (*prepare) (void),
        void (*parent) (void),
        void (*child) (void))
{
  return __register_atfork (prepare, parent, child, &__dso_handle == NULL ? NULL : __dso_handle);
}

As you can see, pthread_atfork() does nothing aside from calling __register_atfork()... so patch that instead!

---

The content of atfork_patch.c now becomes: (using __register_atfork()'s prototype, from here / here)

#include <stdio.h>

int __register_atfork (void (*prepare) (void), void (*parent) (void),
                       void (*child) (void), void *dso_handle) {
  printf("Ignoring pthread_atfork call!\n");
  fflush(stdout);
  return 0;
}

This works for both demos:

$ LD_PRELOAD=./atfork_patch.so ./atfork_demo_main.exec ./atfork_demo1.so
Ignoring pthread_atfork call!
$ LD_PRELOAD=./atfork_patch.so ./atfork_demo_main.exec ./atfork_demo2.so
Ignoring pthread_atfork call!

---