UnmanagedFunctionPointer causes stackoverflow when

Question

I have a simple function inside of a click handler that has a try catch block. If I throw an exception within this try catch block it catches the exception successfully.

If I put a call to an unmanaged DLL before I throw the exception the exception is unhandled and not caught.

What is the unamanged DLL call doing that could be breaking my programs exception handling?

If I run the program in debug mode it catches the exception even with "break on exception" unticked for all exceptions. The application does not crash and runs as expected.

If I run the program as "start without debugging" and hit debug when it crashes I get the following error "Stack cookie instrumentation code detected a stack-based buffer overrun"

edit: It appears the stack overflow breaks the exception handling

I've attached a simplified program that produces the crash.

ISOConnection _comm;  //This is instantiated at another time in the same thread

//C# test function that crashes when run without a debugger attached
bool DoMagic()
{
    try
    {
        //if I uncomment this line the exception becomes unhandled and cannot be caught
        //_comm.ConnectISO15765();

        throw new Exception();
    }
    catch (Exception ex)
    {
        MessageBox.Show("Caught exception")
    }

//Within ISOConnection class
public void ConnectISO15765(){
    ...
    lock(syncLock){
        uint returnCode = J2534Interface.PassThruConnect((uint)DeviceId, (uint)ProtocolID.ISO15765, (uint)ConnectFlag.NONE, (uint)BaudRate.ISO15765, ref ChannelId);


//C# UnmanagedFunctionPointer allocation code
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate uint PassThruConnect(uint deviceId, uint protocolId, uint flags, uint baudRate, ref uint channelId);
public PassThruConnect Connect;

[DllImport("kernel32.dll")]
public static extern IntPtr LoadLibrary(string dllToLoad);

m_pDll = NativeMethods.LoadLibrary(path);
...
pAddressOfFunctionToCall = NativeMethods.GetProcAddress(m_pDll, "PassThruConnect");
if (pAddressOfFunctionToCall != IntPtr.Zero)
    Connect = (PassThruConnect)Marshal.GetDelegateForFunctionPointer(
        pAddressOfFunctionToCall,
        typeof(PassThruConnect));

//C++ function declaration
long PassThruConnect(unsigned long DeviceID, unsigned long ProtocolID, unsigned long Flags, unsigned long Baudrate, unsigned long *pChannelID);

UPDATE

If I replace the call to the UnmanagedFunctionPointer PassThurConnect with the following the crash does NOT occur

[DllImport("op20pt32.dll", EntryPoint = "PassThruConnect", CallingConvention = CallingConvention.Cdecl)]
public static extern uint PassThruConnect2(uint deviceId, uint protocolId, uint flags, uint baudRate, ref uint channelId);

Is there something I am not performing or I am performing incorrectly when assigning the UnmanagedFunctionPointer that would cause the lack of a debugger to create a stackoverflow crash?

What is even stranger is this code appeared to work a few weeks ago. The main changes is the try catch was in another thread and I wasn't using lock(syncLock). Everything is now in one thread however the same crash occurred when run in a BackgroundWorker as well.

UPDATE #2 PROBLEM SEMI-SOLVED

Ok so I rolled back through my commits one by one until it worked. What changed is I went from .NET 3.5 to .NET 4.0

.NET 3.5 does not crash regardless of attaching a debugger or not. .NET 4.0 crashes if a debugger is not attached. To rule out a bug in my code I simply deleted the ConcurrentQueue for my log (the only 4.0 feature I was using) and converted my current code base back to 3.5 and I do not get this error.

To be 100% sure it is an issue with 4.0 I then converted my code base back to 4.0 from 3.5 and left the ConcurrentQueue out (literally just changed the build options and did a rebuild) and the StackOverflow crash is back.

I would prefer to use 4.0, any ideas how to debug this issue?

edit: .NET 4.6.1 also crashes

UPDATE #3 http://codenition.blogspot.com.au/2010/05/pinvokestackimbalance-in-net-40i-beg.html

Apparently pinvokestackimbalance is basically ignored in .NET 3.5, so the problem still exists, it just doesn't crash my application.

Adding the following code to App.Config causes .NET to repair the stack when transitioning back to managed code. A small performance hit but it will fix the problem.

Whilst this does fix the problem, I'd like to know what is wrong with my UnmanagedFunctionPointer to cause the problem in the first place.

<configuration> 
  <runtime> 
    <NetFx40_PInvokeStackResilience enabled="1"/>

Edit: this thread isn't a duplicate, the other one is deleted...

Answer 1

Ok so the problem is the calling convention should be StdCall not Cdecl

This makes sense as the generic J2534 API documentation specifies the following header. Although the header file I was supplied does not make this specification.

extern "C" long WINAPI PassThruConnect
(
unsigned long ProtocolID;
unsigned long Flags
unsigned long *pChannelID
)

Where WINAPI is also known as StdCall not Cdecl like most C/C++ libraries would typically use.

.NET 3.5 allows the wrong calling convention and will "fix" the stack. As of 4.0 this is no longer the case and a PinvokeStackImbalance exception is raised.

You can force 4.0 to also fix the stack with the following code added to your App.Config

<configuration> 
  <runtime> 
    <NetFx40_PInvokeStackResilience enabled="1"/>

Or you can simply fix your calling convention by changing Cdecl to StdCall:

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate uint PassThruConnect(uint deviceId, uint protocolId, uint flags, uint baudRate, ref uint channelID);