Weird MSC 8.0 error: “The value of ESP was not pro

Question

We recently attempted to break apart some of our Visual Studio projects into libraries, and everything seemed to compile and build fine in a test project with one of the library projects as a dependency. However, attempting to run the application gave us the following nasty run-time error message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function pointer declared with a different calling convention.

We have never even specified calling conventions (__cdecl etc.) for our functions, leaving all the compiler switches on the default. I checked and the project settings are consistent for calling convention across the library and test projects.

Update: One of our devs changed the "Basic Runtime Checks" project setting from "Both (/RTC1, equiv. to /RTCsu)" to "Default" and the run-time vanished, leaving the program running apparently correctly. I do not trust this at all. Was this a proper solution, or a dangerous hack?

Answer 1

I saw this error when the code tried to call a function on an object that was not of the expected type.

So, class hierarchy: Parent with children: Child1 and Child2

Child1* pMyChild = 0;
...
pMyChild = pSomeClass->GetTheObj();// This call actually returned a Child2 object
pMyChild->SomeFunction();          // "...value of ESP..." error occurs here

Answer 2

Another case where esp can get messed up is with an inadvertent buffer overflow, usually through mistaken use of pointers to work past the boundary of an array. Say you have some C function that looks like

int a, b[2];

Writing to b[3] will probably change a, and anywhere past that is likely to hose the saved esp on the stack.

Answer 3

I was having this exact same error after moving functions to a dll and dynamically loading the dll with LoadLibrary and GetProcAddress. I had declared extern "C" for the function in the dll because of the decoration. So that changed calling convention to __cdecl as well. I was declaring function pointers to be __stdcall in the loading code. Once I changed the function pointer from __stdcall to__cdecl in the loading code the runtime error went away.

Answer 4

I read this in other forum

I was having the same problem, but I just FIXED it. I was getting the same error from the following code:

HMODULE hPowerFunctions = LoadLibrary("Powrprof.dll");
typedef bool (*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

tSetSuspendState SetSuspendState = (tSuspendStateSig)GetProcAddress(hPowerfunctions, "SetSuspendState");

result = SetSuspendState(false, false, false); <---- This line was where the error popped up. 

After some investigation, I changed one of the lines to:

typedef bool (WINAPI*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

which solved the problem. If you take a look in the header file where SetSuspendState is found (powrprof.h, part of the SDK), you will see the function prototype is defined as:

BOOLEAN WINAPI SetSuspendState(BOOLEAN, BOOLEAN, BOOLEAN);

So you guys are having a similar problem. When you are calling a given function from a .dll, its signature is probably off. (In my case it was the missing WINAPI keyword).

Hope that helps any future people! :-)

Cheers.

Answer 5

I was getting this error calling a function in a DLL which was compiled with a pre-2005 version of Visual C++ from a newer Version of VC (2008). The function had this signature:

LONG WINAPI myFunc( time_t, SYSTEMTIME*, BOOL* );

The problem was that time_t's size is 32 bits in pre-2005 version, but 64 bits since VS2005 (is defined as _time64_t). The call of the function expects a 32 bit variable but gets a 64 bit variable when called from VC >= 2005. As parameters of functions are passed via the stack when using WINAPI calling convention, this corrupts the stack and generates the above mentioned error message ("Run-Time Check Failure #0 ...").

To fix this, it is possible to

#define _USE_32BIT_TIME_T

before including the header file of the DLL or -- better -- change the signature of the function in the header file depending on the VS version (pre-2005 versions don't know _time32_t!):

#if _MSC_VER >= 1400
LONG WINAPI myFunc( _time32_t, SYSTEMTIME*, BOOL* );
#else
LONG WINAPI myFunc( time_t, SYSTEMTIME*, BOOL* );
#endif

Note that you need to use _time32_t instead of time_t in the calling program, of course.

Answer 6

I had this same problem here at work. I was updating some very old code that was calling a FARPROC function pointer. If you don't know, FARPROC's are function pointers with ZERO type safety. It's the C equivalent of a typdef'd function pointer, without the compiler type checking. So for instance, say you have a function that takes 3 parameters. You point a FARPROC to it, and then call it with 4 parameters instead of 3. The extra parameter pushed extra garbage onto the stack, and when it pops off, ESP is now different than when it started. So I solved it by removing the extra parameter to the invocation of the FARPROC function call.