We've been trying to track down some strange behavior in our large legacy app for the past few months. It suffers from random and occasional memory corruption, vtable corruption (??), and other strange and random behavior like infinite loops in std::rbl_tree, std::map, and even std::string s = "abcd".
The target machine is 32-bit Centos 6 so we started with the built-in g++
4.4 but address sanitizer wasn't available, so moved to 4.8 in devtoolset-2 and now have compiled gcc 4.9
from source.
Valgrind
(v3.8.1) doesn't work with any of them - it gives unhandled instruction bytes: 0xC5 0xF9 0x6E 0xC5
AddressSanitizer
reports different errors between gcc 4.8 and 4.9, and often reports bogus global-buffer-overflow errors like
Error reading 0x080e3ad4 where 0x080e3ad4 is located 0 bytes to the right of global variable 'x'
Except that 0x080e3ad4 is the address of x!
Worse yet, the erroneous behavior is different between gcc versions.
I've read a lot of posts comparing the speed advantages of gcc v5, 6, 7, but nothing about stability of the code they produce.
Since it takes a lot of effort to build a new gcc and reconfigure our large app to build with it, and test & package the distribution for customers, here's the question:
- What improvement in stability come with a newer gcc?
- How has
AddressSanitizer
been improved? - Can we expect Valgrind to not barf out on illegal op codes?
All suggestions appreciated. Its been months on this problem.
EDIT
To be clear, I'd like to get instrumentation that works (eg AddressSanitizer
, Valgrind
or others) so we can fix the underlying behavior. Those debug tools working or not is tied to the compiler environment rather than our C++ program. (eg. There is no assembler in the program so Valgrind
should understand the instructions from the compiler and/or the compiler should emit opcodes that are understood by Valgrind, as one example.
EDIT II
G++ compiler flags (for debug compile)
/usr/local/gcc-4.9.4/bin/c++4.9 -c -std=c++98 -m32 -g -ggdb -O0
-Wall -Wextra -Wno-sign-compare -Wcast-align -fdiagnostics-color=auto
-ftemplate-depth=32 -march=native -fPIC -o xx.o xx.cpp
Target hardware: i5-6500