DYLD_LIBRARY_PATH & DYLD_INSERT_LIBRARIES not work

Question

I create a .dylib file and compile it:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

static void* (*real_malloc)(size_t);

void *malloc(size_t size)
{
    void *p = NULL;
    fprintf(stderr, "malloc(%zd) = ", size);
    p = real_malloc(size);
    fprintf(stderr, "%p\n", p);
    return p;
}

void __attribute((constructor))init()
{
    real_malloc = (decltype(real_malloc))dlsym(RTLD_NEXT, "malloc");
    if (NULL == real_malloc) {
        fprintf(stderr, "Error in `dlsym`: %s\n", dlerror());
        return;
    }
}

Then I create a test program that calls malloc. I made sure the call to malloc is not optimized out.

Next I ran the following:

DYLD_PRINT_LIBRARIES=1 X=1 DYLD_INSERT_LIBRARIES=./libTestHook.dylib ./malloctest

It loads it but it does NOT hook the function at all.. Any ideas? I tried this code before El Capitan upgrade and it used to work.. I also made the malloc throw an exception just to see if it was being called. It's not.

What am I missing?

The results were:

sh-3.2# DYLD_PRINT_LIBRARIES=1 X=1 DYLD_INSERT_LIBRARIES=./libTestHook.dylib ./malloctest clear
dyld: loaded: /Users/Brandon/Desktop/./malloctest
dyld: loaded: ./libTestHook.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /usr/lib/system/libcache.dylib
dyld: loaded: /usr/lib/system/libcommonCrypto.dylib
dyld: loaded: /usr/lib/system/libcompiler_rt.dylib
dyld: loaded: /usr/lib/system/libcopyfile.dylib
dyld: loaded: /usr/lib/system/libcorecrypto.dylib
dyld: loaded: /usr/lib/system/libdispatch.dylib
dyld: loaded: /usr/lib/system/libdyld.dylib
dyld: loaded: /usr/lib/system/libkeymgr.dylib
dyld: loaded: /usr/lib/system/liblaunch.dylib
dyld: loaded: /usr/lib/system/libmacho.dylib
dyld: loaded: /usr/lib/system/libquarantine.dylib
dyld: loaded: /usr/lib/system/libremovefile.dylib
dyld: loaded: /usr/lib/system/libsystem_asl.dylib
dyld: loaded: /usr/lib/system/libsystem_blocks.dylib
dyld: loaded: /usr/lib/system/libsystem_c.dylib
dyld: loaded: /usr/lib/system/libsystem_configuration.dylib
dyld: loaded: /usr/lib/system/libsystem_coreservices.dylib
dyld: loaded: /usr/lib/system/libsystem_coretls.dylib
dyld: loaded: /usr/lib/system/libsystem_dnssd.dylib
dyld: loaded: /usr/lib/system/libsystem_info.dylib
dyld: loaded: /usr/lib/system/libsystem_kernel.dylib
dyld: loaded: /usr/lib/system/libsystem_m.dylib
dyld: loaded: /usr/lib/system/libsystem_malloc.dylib
dyld: loaded: /usr/lib/system/libsystem_network.dylib
dyld: loaded: /usr/lib/system/libsystem_networkextension.dylib
dyld: loaded: /usr/lib/system/libsystem_notify.dylib
dyld: loaded: /usr/lib/system/libsystem_platform.dylib
dyld: loaded: /usr/lib/system/libsystem_pthread.dylib
dyld: loaded: /usr/lib/system/libsystem_sandbox.dylib
dyld: loaded: /usr/lib/system/libsystem_secinit.dylib
dyld: loaded: /usr/lib/system/libsystem_trace.dylib
dyld: loaded: /usr/lib/system/libunc.dylib
dyld: loaded: /usr/lib/system/libunwind.dylib
dyld: loaded: /usr/lib/system/libxpc.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libDiagnosticMessagesClient.dylib
A
B
C
D

Answer 1

If you can recompile the executable being hooked, then I think a simpler solution is to recompile the executable with -force_flat_namespace:

➜  clang slow_leak.c -force_flat_namespace -o slow_leak
➜  DYLD_INSERT_LIBRARIES=malloc_hook.dylib ./slow_leak
leaking
in hooked malloc

More info. This is on OSX 10.12.2 MacOS Sierra.

Answer 2

The code in my original post used to work on Yosemite. On El Capitan, it does NOT work. I ended up doing the following approach (DYLD_INTERPOSE + DYLD_INSERT_LIBRARIES):

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>


#define DYLD_INTERPOSE(_replacment,_replacee) \
__attribute__((used)) static struct{ const void* replacment; const void* replacee; } _interpose_##_replacee \
__attribute__ ((section ("__DATA,__interpose"))) = { (const void*)(unsigned long)&_replacment, (const void*)(unsigned long)&_replacee };


void* pMalloc(size_t size) //would be nice if I didn't have to rename my function..
{
   printf("Allocated: %zu\n", size);
   return malloc(size);
}

DYLD_INTERPOSE(pMalloc, malloc);

Answer 3

From dyld docs:

DYLD_INSERT_LIBRARIES This is a colon separated list of dynamic libraries to load before the ones specified in the program. This lets you test new modules of existing dynamic shared libraries that are used in flat-namespace images by loading a temporary dynamic shared library with just the new modules. Note that this has no effect on images built a two-level namespace images using a dynamic shared library unless DYLD_FORCE_FLAT_NAMESPACE is also used.

DYLD_FORCE_FLAT_NAMESPACE Force all images in the program to be linked as flat-namespace images and ignore any two-level namespace bindings. This may cause programs to fail to execute with a multiply defined symbol error if two-level namespace images are used to allow the images to have multiply defined symbols.

So your code needed DYLD_FORCE_FLAT_NAMESPACE=1. Anyway, my implementation works for /usr/local/bin/git only with this option enabled.