How to deal with the Unicode characters in C++

Question

We have a commenting system built into our engine that allows programmers to put comments for various exposed variables/objects which are then used by the GUI front-end for tool-tips and help.

Recently, certain tool-tips started crashing, and after much wasted time I tracked it down to the the character: ’ which, unless I am mistaken, is a unicode character and not available in ASCII.

Taking this answer into consideration, I assumed wstring would fix the problem. Before making changes in the bigger project, I created a test project to see if wstring would solve the issue. Although the project doesn't crash, the behavior is not as expected for wstring.

#include <iostream>
#include <string>

using namespace std;

int main()
{
    string someString = "successive attack that DOESN’T result";
    wstring someWString = L"successive attack that DOESN’T result";

    cout << someString << endl;
    wcout << someWString << endl;

    return 0;
}

//Console Output//
successive attack that DOESNÆT result 
successive attack that DOESNPress any key to continue . . .

I read this article quite some time ago and thought I understood the problems associated with character sets, but that is obviously not the case. I would appreciate a solution to this problem as well as a good explanation of what is happening and how to avoid problems similar to this in the future.

Answer 1

Since you are using Visual Studio I assume you are using Windows. The Windows console does not support unicode. It uses the OEM char set. You can convert between the two using CharToOemW/OemToCharW. Obviously it will not be able to represent all unicode characters.

Windows uses UTF16 for its system API. If your tooltips uses the Windows API it is probably wstring that you want to use. However, you can use UTF8 instead and convert this to UTF16 before calling the Windows API. This conversion can be performed using MultiByteToWideChar/WideCharToMultiByte.

Answer 2

Since you are dealing with Unicode characters, it would be appropriate if you set Character Set to Use Unicode Character Set in projects properties.

Another possible problem could be the encoding of source files. The best practice while working with Unicode characters is to have your source files encoded in UTF-8, especially files where you define string literals like this one. Note that UTF-8 without BOM could be troublesome because Visual Studio needs this BOM so that it can intepret files content properly. Convert your files (I use Notepad++ for this) and convert it so that they are encoded in UTF-8