I have a DLL file library.dll which contains a function foo. The function foo calls a WinAPI function goo. I wrote an application that calls foo from library.dll. The problem is that I want to override the call to goo function by my own function hoo I declared in the application (not in the DLL).
How can I hook the call to goo function? I'm not looking for a global hook, I just want to override calls made by application I wrote.
There is library called Detours provided by Microsoft Research: http://research.microsoft.com/en-us/projects/detours/. You can use it to re-route any API call in Windows.
It does exactly what you describe -- instead of calling into Win32 API, your function gets called. Within that function you are free to do what you want, e.g. you can call again to the original Win32 function or you can return failure code right away or anything you like.
Express edition of Detours is free, but it is limited for non-commercial use on x86 architecture.
Patch the import descriptor for goo
in library.dll
's import address table. IAT patching is a well known hooking technique for intercepting function calls between two PE modules.