Crash caused by invalid exception unwinding?

I'm debugging a crash which occurs in one of our tests. We have an ODBC driver .so written in C++, which is being tested via iODBC using our test tool (called 'Touchstone'), which is also written in C++.

I've compiled all three on Solaris 10 (x86) in 64-bit mode using the Oracle Solaris Studio 12.4 (NOT GCC).

The crash only occurs when Touchstone is built in release mode, so I've spent a fair bit of time stepping through assembly in dbx, and what seems to happen is the following:

The 'this' pointer of the 'statement' object is stored into r14
We call into SQLColAttributes in the driver manager (DM), which calls into SQLColAttributeW in the driver.
Before returning into the DM, it spills r14 to the stack in one of SQLColAttributeW's callees.
Farther down the stack, an exception is thrown (which eventually is caught, before leaving the driver and returning into the DM), which seems to 'mess up' during stack unwinding, so that once I finally return into touchstone, r14 is garbage. Touchstone attempts to retrieve a member from the statement object and very quickly crashes with a SEGV.

One of the first things I did when I got to #3 was to put a hardware write watch on the stack where the register was getting spilled to, but it doesn't get hit until after returning into touchstone, after the register has already been corrupted.

Then I noticed that the code which seem to cause the corruption was throwing an exception, and remembered this, which I have encountered before, when using Touchstone built with solaris studio use a driver built with gcc, so I recompiled iODBC with solaris studio, and running ldd shows no dependency on it anymore, but it's still crashing in the same way.

I've also tried the workaround suggested in that article (LD_PRELOAD=/usr/sfw/lib/amd64/libgcc_s.so), but that didn't change anything either.

I was also able to compile valgrind, and it seems to support my hypothesis:

-bash-4.1$ $VG --tool=memcheck $TC -te ApiTestEnv_utf32.xml -ts ApiTestSuite.xml -o crash -rts COLATTRIBUTETESTS
==900== Memcheck, a memory error detector
==900== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==900== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==900== Command: /export/home/bamboo/Touchstone -te ApiTestEnv_utf32.xml -ts ApiTestSuite.xml -o crash -rts COLATTRIBUTETESTS
==900==
Simba Test Verbose Log Started on Thu Jul 14 11:35:25 2016

Touchstone test utility for ODBC and OLE DB for OLAP
Version: 4.5.0.5 (64-bit)
Copyright (c) 2012 Simba Technologies Incorporated

Starting test run
---------------------------
        API Tests: COLATTRIBUTETESTS: SQLCOLATTRIBUTES_ERROR (1)==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F94FF6D2: __1cFSimbaHSupportbAIniFileConfigurationReaderLOpenIniFile6Mrkn0BNsimba_wstring_5_pn0BITextFile__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEBC3: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0DDmap4n0E_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______n0G_n0DJallocator4n0DEpai
r4C5n0J_______r58p5_b_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEA51: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______r58p5b_b_ (in /export/home/bamboo/sol
-crash/libInternalTest_debug.so)
==900==    by 0x7F950CD97: __1cFSimbaHSupportSSimbaSettingReaderUInternal_ReadSetting6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950C2C6: __1cFSimbaHSupportSSimbaSettingReaderLReadSetting6FrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950A2DC: __1cFSimbaHSupportSSimbaSettingReaderSGetAppCharEncoding6F_n0BMEncodingType__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8E29C: __1cFSimbaDDSIJDSIDriverYSetDefaultPropertyValues6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8D787: __1cFSimbaDDSIJDSIDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57748: __1cFSimbaMInternalTestIITDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F86D7330: __1cFSimbaDDSIQDSIDriverFactory6FrL_pn0BHIDriver__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8E17FEB: __1cFSimbaDDSIWSharedSingletonManagerKInitialize6Fb_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==  Address 0x37fe5ba8 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F94FEBC3: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0DDmap4n0E_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______n0G_n0DJallocator4n0DEpai
r4C5n0J_______r58p5_b_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEA51: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______r58p5b_b_ (in /export/home/bamboo/sol
-crash/libInternalTest_debug.so)
==900==    by 0x7F950CD97: __1cFSimbaHSupportSSimbaSettingReaderUInternal_ReadSetting6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950C2C6: __1cFSimbaHSupportSSimbaSettingReaderLReadSetting6FrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950A2DC: __1cFSimbaHSupportSSimbaSettingReaderSGetAppCharEncoding6F_n0BMEncodingType__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8E29C: __1cFSimbaDDSIJDSIDriverYSetDefaultPropertyValues6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8D787: __1cFSimbaDDSIJDSIDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57748: __1cFSimbaMInternalTestIITDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F86D7330: __1cFSimbaDDSIQDSIDriverFactory6FrL_pn0BHIDriver__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8E17FEB: __1cFSimbaDDSIWSharedSingletonManagerKInitialize6Fb_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0644BD: __1cFSimbaEODBCGDriverUInitializeSingletons6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==  Address 0x37fe5bd8 is on thread 1's stack
==900==  856 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F7B58945: __1cFSimbaMInternalTestIITDriverbAInitializeUnicodeStringMap6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57C96: __1cFSimbaMInternalTestIITDriverRCreateEnvironment6M_pn0ADDSIMIEnvironment__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA06087D: __1cFSimbaEODBCGDriverRCreateEnvironment6Mppv_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB138F: SQLAllocHandle (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE21BE90: _iodbcdm_driverload (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE21FDB0: SQLDriverConnect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE220DE3: SQLDriverConnectW (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69132: __1cFSimbaIODBCTestDCliRSqlDriverConnectW6Mpv3pwh4hphH_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD9CDB: __1cFSimbaIODBCTestKConnectionRSqlDriverConnectW6MpvpkwhpwhphHrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xABC586: __1cFSimbaIODBCTestRConnectionFactorySMakeConnectionInC46Fpn0BLEnvironment_rkn0BHODBCStr__pn0BKConnection__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD2F39: __1cFSimbaIODBCTestSOdbcTestCaseBaseS1MexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fe6798 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F7B57C96: __1cFSimbaMInternalTestIITDriverRCreateEnvironment6M_pn0ADDSIMIEnvironment__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA06087D: __1cFSimbaEODBCGDriverRCreateEnvironment6Mppv_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB138F: SQLAllocHandle (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE21BE90: _iodbcdm_driverload (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE21FDB0: SQLDriverConnect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE220DE3: SQLDriverConnectW (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69132: __1cFSimbaIODBCTestDCliRSqlDriverConnectW6Mpv3pwh4hphH_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD9CDB: __1cFSimbaIODBCTestKConnectionRSqlDriverConnectW6MpvpkwhpwhphHrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xABC586: __1cFSimbaIODBCTestRConnectionFactorySMakeConnectionInC46Fpn0BLEnvironment_rkn0BHODBCStr__pn0BKConnection__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD2F39: __1cFSimbaIODBCTestSOdbcTestCaseBaseS1MexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD228AB: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBaseMexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fe67c8 is on thread 1's stack
==900==  856 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA039B3D: __1cFSimbaEODBCKDescriptorOGetHeaderField6kMhpvpi_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA11C104: __1cFSimbaEODBCOStatementStateQSQdDLNumResultCols6Mph_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA10903D: __1cFSimbaEODBCJStatementQSQdDLNumResultCols6Mph_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F52DE0: __1cGDoTask4nFSimbaEODBCUSQdDLNumResultColsTask__6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9ED63DB: SQLNumResultCols (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23C4E1: _iodbcdm_NumResultCols (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE223169: _iodbcdm_do_cursoropen (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224295: SQLExecDirect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224537: SQLExecDirect (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69B5D: __1cFSimbaIODBCTestDCliNSqlExecDirect6MpvpCi_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE39BA: __1cFSimbaIODBCTestJStatementNSqlExecDirect6MpkCirkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee258 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA10903D: __1cFSimbaEODBCJStatementQSQdDLNumResultCols6Mph_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F52DE0: __1cGDoTask4nFSimbaEODBCUSQdDLNumResultColsTask__6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9ED63DB: SQLNumResultCols (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23C4E1: _iodbcdm_NumResultCols (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE223169: _iodbcdm_do_cursoropen (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224295: SQLExecDirect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224537: SQLExecDirect (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69B5D: __1cFSimbaIODBCTestDCliNSqlExecDirect6MpvpCi_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE39BA: __1cFSimbaIODBCTestJStatementNSqlExecDirect6MpkCirkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD205EA: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee288 is on thread 1's stack
==900==  1128 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA039B3D: __1cFSimbaEODBCKDescriptorOGetHeaderField6kMhpvpi_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA11F45D: __1cFSimbaEODBCOStatementStateXDoColAttributeOnlyCount6MHphpl_nDstdEpair4Cp2Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA12962D: __1cFSimbaEODBCWStatementStateExecutedQSQdDLColAttributeW6MHHpvhphpl_nDstdEpair4Cpn0BOStatementState_Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fedf48 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA12962D: __1cFSimbaEODBCWStatementStateExecutedQSQdDLColAttributeW6MHHpvhphpl_nDstdEpair4Cpn0BOStatementState_Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22371: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebEVerifyColAttributesNumberField6Mhl_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD20720: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fedf78 is on thread 1's stack
==900==  1128 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee258 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee288 is on thread 1's stack
==900==  776 bytes below stack pointer
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE161A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE1634: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE1224: __1cFSimbaIODBCTestGHandleDlog6kM_pn0AETestNVerboseLogger__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE1827: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Invalid read of size 1
==900==    at 0x7FE3AD4FD: mutex_lock_impl (in /lib/amd64/libc.so.1)
==900==    by 0x7FE3AD793: mutex_lock (in /lib/amd64/libc.so.1)
==900==    by 0x1317140: __1cFSimbaETestNVerboseLoggerDLog6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE184A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==  Address 0x650000017b is not stack'd, malloc'd or (recently) free'd
==900==
==900==
==900== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==900==  Access not within mapped region at address 0x650000017B
==900==    at 0x7FE3AD4FD: mutex_lock_impl (in /lib/amd64/libc.so.1)
==900==    by 0x7FE3AD793: mutex_lock (in /lib/amd64/libc.so.1)
==900==    by 0x1317140: __1cFSimbaETestNVerboseLoggerDLog6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE184A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==  If you believe this happened as a result of a stack
==900==  overflow in your program's main thread (unlikely but
==900==  possible), you can try to increase the size of the
==900==  main thread stack using the --main-stacksize= flag.
==900==  The main thread stack size used in this run was 8388608.
==900==
==900== HEAP SUMMARY:
==900==     in use at exit: 19,994,466 bytes in 157,661 blocks
==900==   total heap usage: 416,854 allocs, 259,193 frees, 56,010,824 bytes allocated
==900==
==900== LEAK SUMMARY:
==900==    definitely lost: 0 bytes in 0 blocks
==900==    indirectly lost: 0 bytes in 0 blocks
==900==      possibly lost: 10,903,944 bytes in 79,967 blocks
==900==    still reachable: 9,090,522 bytes in 77,694 blocks
==900==         suppressed: 0 bytes in 0 blocks
==900== Rerun with --leak-check=full to see details of leaked memory
==900==
==900== For counts of detected and suppressed errors, rerun with: -v
==900== Use --track-origins=yes to see where uninitialised values come from
==900== ERROR SUMMARY: 14 errors from 14 contexts (suppressed: 0 from 0)
Segmentation Fault

searching for _Uwn_jmp gives me this unfortunately (look at the name), which means that I was seeing the same thing for what I dismissed as gcc/cc incompatibility.

The same tests work fine for 32-bit.

Any ideas?

(not a full answer, but this seemed too long for comments).

TL:DR: check the ABI instead of guessing about which part is wrong.

r14 is a call-preserved register (and not used for arg-passing) in the SysV AMD64 ABI. Functions that want to use it will push it in their prologue and pop it in their epilogue. But they don't care what their caller was using it for. They just have to make sure it has the same value on return as it did on entry.

The register state on entry to a catch clause seems to be documented in Section 6.2.6 of the ABI:

Transferring Control to a Landing Pad

...

Prior to executing code in the landing pad, the unwind library restores registers not altered by the personality routine, using the context record, to their state in that frame before the call that threw the exception, as follows. All registers specified as callee-saved by the base ABI are restored, [and %rsi, %rdi, %rdx, %rcx are used to pass info about the exception].

The original end of that last sentence in the ABI is confusingly worded. My paraphrase inside [ ] may be wrong. (See the last paragraph of that subsection).

Based on this and previous sections (describing how cleanup code for each frame is called even if it doesn't have a catch), I'm almost certain that call-preserved registers are restored in a catch. This is the same mechanism that runs destructors during unwinding, which is necessary.

So it sounds like the error is that r14 isn't being restored correctly. The function that catches the exception and tries to use r14 isn't doing anything wrong.

Maybe try including destructors that print some logging info in some of the functions that will have their stack frame unwound? Maybe have them check a global to decide whether to print, so you can set a global and then make the call that will result in an exception being thrown, so you don't get a screen full of log messages from calls that don't throw.

Maybe destructors/cleanup aren't being called correctly during phase2 of unwinding, if you didn't see r14 being restored.

... put a hardware write watch on the stack where the register was getting spilled to, but it doesn't get hit until after returning into touchstone

You should have been using a read watchpoint to look for it being read during the unwind cleanup code for the function that spilled it. It's not surprising that nothing overwrites the spill location until after entering the catch clause.