Can the DllMain of an .exe be called?

Question

My question is not exactly the same as this one (it's not theoretical, there is only a main thread without message loop, InitInstance and ExitInstance are no fitting calls).

I'm using a console app without message loop; this app loads an exe with the LoadLibrary function, so that it can use its exported functions. Bad news: the DllMain function of the exe is not called (and I verified the symbols tables, using a def file, DllMain appears correctly); the doc says it's called if the loaded module is a DLL (too bad).

What are the conditions (if they exist) which could lead to the execution of the exe's DllMain function when LoadLibrary is called (and maybe again when FreeLibrary is called)?

Best regards

Answer 1

Completing the good answer of MSalters:

So, then, call the "fake" DllMain with DLL_XXX_ATTACH just after LoadLibrary and with DLL_XXX_DETACH just before FreeLibrary, and make manually the other calls.

Another following implementation would be to build and load an interface DLL which could callback automatically the EXE on its fake DllMain (I don't know if it could work); but it may be more complicated than just manually calling the fake DllMain in a number of cases. (can't LoadLibrary in a DllMain)

Answer 2

Indeed the Name "DllMain" of the function is ignored by Windows completely (the old Windows API documentation of Windows NT 3.x stated this explicitly).

When a DLL is loaded not the function DllMain() but the function that is located at the file's entry point is called.

Of course the linker will create the DLL file in a way that DllMain() is this function.

However for EXE files the entry function (that will call WinMain()) is located at the entry point.

So it is obvious that Windows cannot call this function when loading the EXE file as DLL.

Answer 3

The conditions are:

1) Binary being loaded was compiled as DLL (when using gcc/ld it means using --shared option; If you use --shared, resulting file will be a dll, and will not run, see below)

2) IMAGE_FILE_DLL is set in PE file header of the binary file being loaded. If it is set, file is a dll, and Windows linker will call its DllMain() function for you when it links this file to your program (doesn't matter how it's linked - LoadLibrary() at runtime or -llibraryname at compile time). For that the file must also satisfy (1). But with this flag the binary file being loaded will not be runnable. If IMAGE_FILE_DLL is not set, DllMain() will not be called when file is loaded into your program.

Compiling dll with --shared and then manually removing IMAGE_FILE_DLL from its header (i.e. by using hex editor) will not work - when you run it, only DllMain() will be executed, and fdwReason will be a non-defined number (0x28ffd4 on my machine).

Update

All DLL and EXE files on Windows are PE files, the difference is how they are linked, and which flags are set in their headers. That is why i write file being loaded, not dll being loaded.

The last paragraph also describes the scenario where you compile the file as dll, and then turn it into exe by messing with its header. It doesn't work.

Naming has nothing to do with it (you can choose any name, and with some pexports+dlltool tinkering you can create an import library for an .exe file and be able to link it as -lexenamewithoutextension

To clarify:

• if you compile it without --shared:
  • IMAGE_FILE_DLL will not be set in it, it will be runnable, but DllMain() will NOT be called when you link it.
• if you compile it with --shared:
  • IMAGE_FILE_DLL will be set in it, it will NOT be runnable, but DllMain() will be called when you link it.
• if you compile it without --shared, then switch on the IMAGE_FILE_DLL flag in it manually:
  • it will NOT be runnable anymore, and no DllMain() will be called when you link it.
• if you compile it with --shared, then switch off the IMAGE_FILE_DLL flag in it manually:
  • it will be runnable, but DllMain() will be executed instead of main(), and DllMain() will NOT be called when you link it.

Answer 4

The most obvious condition is that the process calling LoadLibrary() explicitly gets GetProcAddress("DllMain") and then calls it.