Forcing name mangling in x64 DLL

Question

I am porting a 32-bit application to 64-bit. The application supports plugins which are DLLs. Unfortunately, one of the mandatory functions each plugin needs to have is called FreeLibrary which of course conflicts with the kernel32 API of the same name. The reason why my plugin API uses the FreeLibrary name is that the application originated on a different platform where FreeLibrary doesn't clash with any OS APIs.

However, even on 32-bit Windows using FreeLibrary is not a problem because 32-bit DLLs use name mangling, i.e. the function is stored as _FreeLibrary and hence doesn't clash with the kernel32 API.

On 64-bit, though, I have a problem now because 64-bit doesn't seem to use name mangling. On 64-bit the compiler creates a symbol named FreeLibrary and this of course clashes with the kernel32 API of the same name and refuses to link, leading to the following error:

Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

kernel32.lib(KERNEL32.dll) : error LNK2005: FreeLibrary already defined in test.o
    Creating library test.lib and object test.exp
test.dll : fatal error LNK1169: one or more multiply defined symbols found

Thus, I'm wondering whether there is any way to force x64 DLLs to use name mangling as on 32-bit so that my DLL can export a symbol named FreeLibrary without any clashes with kernel32?

Or is there any other solution to work around this problem?

The only solution I see is to rename FreeLibrary to something that doesn't clash with OS APIs for the x64 versions of my application but of course I'd like to avoid this because it reduces the consistency of my application's plugin API. If possible, I'd like to keep the FreeLibrary name across all platforms and architectures.

Any ideas? I guess if it was possible on 32-bit to use reserved names like FreeLibrary in a DLL then there should be a way to do it on 64-bit as well, shouldn't there? But I don't see how...

Answer 1

I would not try to convince the compiler to mangle the names of the functions. That way lies madness.

To clarify, your only issue is the duplicate symbols when linking plugin DLLs. The application itself shouldn't care what the functions are named, because it's going to call the plugin entry points through function pointers received via GetProcAddress.

Assuming, the plugins don't ever need to call Windows APIs implemented in kernel32.dll, you could try omitting kernel32.dll from the linker command. (See the /NODEFAULTLIB option.) If kernel32.dll isn't part of the link, there should be no collision.

But that won't work because /MT in the command line implies you depend on the C run-time library, which, in turn, depends on some kernel32.dll APIs. (Also, are you sure you want to link the plugins against the static run-time library and not the DLL version?)

So the remaining option is to change the name of the function. This should be trivial, since you're porting anyway. You could even use the preprocessor to hack the name when compiling the plugins so that you don't need to change their source code:

cl /EHsc /c /DFreeLibrary=Plugin_FreeLibrary /Fotest.o test.c

And then change the GetProcAddress call in the application to look for Plugin_FreeLibrary instead of FreeLibrary, which I assume happens in just one place.