Application hung during CoCreateInstance when usin

Question

When I don't configure a client DLL (globalsd.dll) to use Isolated COM to access a .NET assembly that exposes a COM interface (FSCulture.dll) before running the main executable (FCMTSYSM.EXE) I get an application exception (Access Violation) as the CoCreateInstance fails during the static initialization of the DLL (syslev0.dll) that calls the client DLL (globalsd.dll) during startup (the code continues to try to run and use the non-existent COM object). This much I understand and can deal with. I need to improve the error handling and/or register the COM object with regasm and/or configure globalsd.dll for Isolated COM.

However, when I configure globalsd.dll for Isolated COM, giving it a TLB file and the FSCulture.DLL filename, the whole application just hangs during CoCreateInstance in globalsd.dll, and I can't figure out where it's wandered off to. I tried setting a breakpoint in the constructor of the object being created in FSCulture.dll but it never gets hit. It's like I need to debug the internals of the Isolated COM framework. What's the best way to investigate and fix this?

After loading symbols from the symbol server, the call stack might make a little more sense, but it's still not especially helpful to me:

ntdll.dll!_ZwDelayExecution@8() Unknown
ntdll.dll!__LdrpInitialize@8()  Unknown
ntdll.dll!_LdrInitializeThunk@8()   Unknown

As a test I tried to create a VB6 client EXE using the same COM server with isolated COM and ran into a different problem. I'm getting

Run-time error '-2146234304 (80131040)': Automation error

When I rename the server .manifest file and re-run the client, I get the same error. Then if I touch the client manifest (no change to the file content, just re-save the same file) the error changes to:

The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.

I run sxstrace to see what's going on, and not surprisingly, it can't find the server manifest file, so it errors out. Now if I restore the server manifest file to its original name and re-run the client, I'm back to getting error 80131040. And if I run sxstrace in this situation, the parsed output file is empty.

At this point if I run regasm to register the server, I still get error 80131040. But if I rename the client manifest, the client test runs fine.

I don't understand why just touching the manifest file (making no change) has any effect. I also don't understand why SXSTrace is outputting nothing in cases where I get the 80131040 error. The client manifest looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
   <assemblyIdentity type="win32" name="Project1" version="1.0.0.0" />
   <dependency>
   <dependentAssembly>
   <assemblyIdentity type="win32" name="SoftBrands.FourthShift.FSCulture" version="1.0.0.0" />
   </dependentAssembly>
   </dependency>
</assembly>

The server manifest looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" 
  manifestVersion="1.0">
<assemblyIdentity type="win32" name="SoftBrands.FourthShift.FSCulture" version="1.0.0.0" />
<clrClass
            clsid="{23D4FF3D-EEDF-4F68-AD65-749958EE3B2A}"
            progid="SoftBrands.FourthShift.FSCulture.FSCulture"
            threadingModel="Both"
            name="SoftBrands.FourthShift.FSCulture.FSCulture" >
</clrClass>
</assembly>

Edit I realized that there must be another thread with a more helpful call stack in my original scenario, so I tried again. It turns out there is. This is the call stack on the main thread:

ntdll.dll!_ZwAlpcSendWaitReceivePort@32()   Unknown
rpcrt4.dll!LRPC_CASSOCIATION::AlpcSendWaitReceivePort(unsigned long,struct _PORT_MESSAGE *,struct _ALPC_MESSAGE_ATTRIBUTES *,struct _PORT_MESSAGE *,unsigned long *,struct _ALPC_MESSAGE_ATTRIBUTES *,union _LARGE_INTEGER *)   Unknown
rpcrt4.dll!LRPC_BASE_CCALL::DoSendReceive(void) Unknown
rpcrt4.dll!LRPC_BASE_CCALL::SendReceive(struct _RPC_MESSAGE *)  Unknown
rpcrt4.dll!_I_RpcSendReceive@4()    Unknown
rpcrt4.dll!_NdrSendReceive@8()  Unknown
rpcrt4.dll!@NdrpSendReceive@4() Unknown
rpcrt4.dll!_NdrClientCall2()    Unknown
ole32.dll!ServerAllocateOXIDAndOIDs(void * hServer, void * phProcess, unsigned __int64 * poxidServer, long fApartment, unsigned long cOids, unsigned __int64 * aOid, unsigned long * pcOidsAllocated, tagOXID_INFO * poxidInfo, tagDUALSTRINGARRAY * pdsaStringBindings, tagDUALSTRINGARRAY * pdsaSecurityBindings, unsigned __int64 * pdwOrBindingsID, tagDUALSTRINGARRAY * * ppdsaOrBindings) Line 246    C
ole32.dll!CRpcResolver::ServerRegisterOXID(tagOXID_INFO & oxidInfo, unsigned __int64 * poxid, unsigned long * pcOidsToAllocate, unsigned __int64 * arNewOidList) Line 1028  C++
ole32.dll!OXIDEntry::RegisterOXIDAndOIDs(unsigned long * pcOids, unsigned __int64 * pOids) Line 1303    C++
ole32.dll!OXIDEntry::AllocOIDs(unsigned long * pcOidsAlloc, unsigned __int64 * pOidsAlloc, unsigned long cOidsReturn, unsigned __int64 * pOidsReturn)   C++
ole32.dll!CComApartment::CallTheResolver() Line 639 C++
ole32.dll!CComApartment::InitRemoting() Line 996    C++
ole32.dll!CComApartment::StartServer() Line 1204    C++
ole32.dll!InitChannelIfNecessary() Line 1021    C++
ole32.dll!CDllHost::GetApartmentToken(unsigned long & hActivator) Line 461  C++
ole32.dll!DoSTMTApartmentCreate(unsigned long & hActivator) Line 211    C++
ole32.dll!CClassCache::GetActivatorFromDllHost(int fSixteenBit, unsigned long dwDllThreadModel, unsigned long * phActivator)    C++
ole32.dll!CClassCache::GetOrCreateApartment(const ACTIVATION_PROPERTIES & ap, DLL_INSTANTIATION_PROPERTIES * pdip, unsigned long * phActivator) Line 4892   C++
ole32.dll!FindOrCreateApartment(const _GUID & Clsid, unsigned long actvflags, DLL_INSTANTIATION_PROPERTIES * pdip, unsigned long * phActivator) Line 2653   C++
ole32.dll!CProcessActivator::GetApartmentActivator(ActivationPropertiesIn * pInActProperties, ISystemActivator * * ppActivator) Line 1214   C++
ole32.dll!CProcessActivator::CCICallback(unsigned long dwContext, IUnknown * pUnkOuter, ActivationPropertiesIn * pActIn, IActivationPropertiesIn * pInActProperties, IActivationPropertiesOut * * ppOutActProperties) Line 1703 C++
ole32.dll!CProcessActivator::AttemptActivation(ActivationPropertiesIn * pActIn, IUnknown * pUnkOuter, IActivationPropertiesIn * pInActProperties, IActivationPropertiesOut * * ppOutActProperties, HRESULT (unsigned long, IUnknown *, ActivationPropertiesIn *, IActivationPropertiesIn *, IActivationPropertiesOut * *) * pfnCtxActCallback, unsigned long dwContext) Line 1630   C++
ole32.dll!CProcessActivator::ActivateByContext(ActivationPropertiesIn * pActIn, IUnknown * pUnkOuter, IActivationPropertiesIn * pInActProperties, IActivationPropertiesOut * * ppOutActProperties, HRESULT (unsigned long, IUnknown *, ActivationPropertiesIn *, IActivationPropertiesIn *, IActivationPropertiesOut * *) * pfnCtxActCallback) Line 1487    C++
ole32.dll!CProcessActivator::CreateInstance(IUnknown * pUnkOuter, IActivationPropertiesIn * pInActProperties, IActivationPropertiesOut * * ppOutActProperties) Line 1377    C++
ole32.dll!ActivationPropertiesIn::DelegateCreateInstance(IUnknown * pUnkOuter, IActivationPropertiesOut * * ppActPropsOut) Line 1917    C++
ole32.dll!CClientContextActivator::CreateInstance(IUnknown * pUnkOuter, IActivationPropertiesIn * pInActProperties, IActivationPropertiesOut * * ppOutActProperties) Line 685   C++
ole32.dll!ActivationPropertiesIn::DelegateCreateInstance(IUnknown * pUnkOuter, IActivationPropertiesOut * * ppActPropsOut) Line 1917    C++
ole32.dll!ICoCreateInstanceEx(const _GUID & Clsid, IUnknown * punkOuter, unsigned long dwClsCtx, _COSERVERINFO * pServerInfo, unsigned long dwCount, unsigned long dwActvFlags, tagMULTI_QI * pResults, ActivationPropertiesIn * pActIn) Line 1334  C++
ole32.dll!CComActivator::DoCreateInstance(const _GUID & Clsid, IUnknown * punkOuter, unsigned long dwClsCtx, _COSERVERINFO * pServerInfo, unsigned long dwCount, tagMULTI_QI * pResults, ActivationPropertiesIn * pActIn) Line 332  C++
ole32.dll!CoCreateInstanceEx(const _GUID & Clsid, IUnknown * punkOuter, unsigned long dwClsCtx, _COSERVERINFO * pServerInfo, unsigned long dwCount, tagMULTI_QI * pResults) Line 157    C++
ole32.dll!CoCreateInstance(const _GUID & rclsid, IUnknown * pUnkOuter, unsigned long dwContext, const _GUID & riid, void * * ppv) Line 110  C++

Answer 1

There are many pitfalls when attempting to set up isolated COM between an un-managed client and a managed server. One of them is the fact that the isolated COM settings are not intended for this scenario (I believe they are for referencing an un-managed server). I suspect another is that trying to debug this configuration without having the proper manifest files embedded can result in different results between release mode and debug mode because of where, how and when manifest files are generated. All the answers for fixing the scenario in this question are now covered at https://stackoverflow.com/a/34076433/78162.

Update: More details about the root cause of this issue have been discovered after discussions with Microsoft on related issue Application hanging during CoCreateInstance of .NET-based COM object

Namely, one must be careful when creating static instances of objects that require dynamic initialization in a DLL because those will run during DllMain if the /clr compiler switch is not applied to the source file, and DllMain can cause deadlocks if managed code is referenced within its scope due to loader locks documented in more detail at https://msdn.microsoft.com/en-us/library/ms173266(v=vs.120).aspx

One simple solution (the one I used when re-encountering this issue on another context) is to apply the /clr compiler switch to the source file that had a static instance of an object whose constructor referenced a managed-code COM object.