How to install a DebugBreak handler?

Question

We are setting up Appveyor for our Visual Studio solution, which produces a C++ library. A few of our tests [dumb] fuzz C++ objects to ensure they don't do something unexpected. Under debug builds it causes an assert to fire (and in release builds it just throws).

We use a custom assert to avoid Posix behavior of crashing a program being debugged. It is shown below. It appears Appveyor or the Operating System kills the program if an assert fires and a debugger is not attached:

We want to install a DebugBreak handler if a debugger is not present. This should confirm its the OS doing the killing. Ideally, the handler will work from Windows XP onwards and VS2002 and above (those are the Windows combinations we support).

How do we install a DebugBreak handler on Windows platforms?

---

#  define MYLIB_ASSERT(exp) {                                     \
    if (!(exp)) {                                                 \
      std::ostringstream oss;                                     \
      oss << "Assertion failed: " << (char*)(__FILE__) << "("     \
          << (int)(__LINE__) << "): " << (char*)(__FUNCTION__)    \
          << std::endl;                                           \
      std::cerr << oss.str();                                     \
      __debugbreak();                                             \
    }                                                             \
}

---

We can't really tell who is responsible at the because the behavior is not documented on MSDN at DebugBreak and __debugbreak or C/C++ Assertions.

Answer 1

We want to install a DebugBreak handler if a debugger is not present.

you need install usual windows exception handler, by using __try / __except or by AddVectoredExceptionHandler or by SetUnhandledExceptionFilter

when int 3 instruction is executed in your app (by calling __debugbreak or DebugBreak) - and debugger not present or not handle this - your exception handler will be called with exception code STATUS_BREAKPOINT.

Answer 2

There are a few distinct ways to handle this.

From the process that launches the one that may call DebugBreak(), you can use WaitForDebugEvent (or WaitForDebugEventEx) and ContinueDebugEvent to handle debug events from the child. I.e., the parent acts as a debugger, and the child as a debuggee, similar to how Visual Studio (among many others) debugger works.

You can also attach to a running process using DebugActiveProcess. After you've attached, most of debugging is similar to a parent debugging its child process.

If you can't (or don't want to) do either of those, you can install a post-mortem debugger. You do this by specifying the debugger in the registry, as described on MSDN. Windows has a "Windows Error Reporting" (WER), which invokes the specified post-mortem debugger.