Output unicode strings in Windows console app

Question

Hi I was trying to output unicode string to a console with iostreams and failed.

I found this: Using unicode font in c++ console app and this snippet works.

SetConsoleOutputCP(CP_UTF8);
wchar_t s[] = L"èéøÞǽлљΣæča";
int bufferSize = WideCharToMultiByte(CP_UTF8, 0, s, -1, NULL, 0, NULL, NULL);
char* m = new char[bufferSize]; 
WideCharToMultiByte(CP_UTF8, 0, s, -1, m, bufferSize, NULL, NULL);
wprintf(L"%S", m);

However, I did not find any way to output unicode correctly with iostreams. Any suggestions?

This does not work:

SetConsoleOutputCP(CP_UTF8);
utf8_locale = locale(old_locale,new boost::program_options::detail::utf8_codecvt_facet());
wcout.imbue(utf8_locale);
wcout << L"¡Hola!" << endl;

EDIT I could not find any other solution than to wrap this snippet around in a stream. Hope, somebody has better ideas.

//Unicode output for a Windows console 
ostream &operator-(ostream &stream, const wchar_t *s) 
{ 
    int bufSize = WideCharToMultiByte(CP_UTF8, 0, s, -1, NULL, 0, NULL, NULL);
    char *buf = new char[bufSize];
    WideCharToMultiByte(CP_UTF8, 0, s, -1, buf, bufSize, NULL, NULL);
    wprintf(L"%S", buf);
    delete[] buf; 
    return stream; 
} 

ostream &operator-(ostream &stream, const wstring &s) 
{ 
    stream - s.c_str();
    return stream; 
} 

Answer 1

The wcout must have the locale set differently to the CRT. Here's how it can be fixed:

int _tmain(int argc, _TCHAR* argv[])
{
    char* locale = setlocale(LC_ALL, "English"); // Get the CRT's current locale.
    std::locale lollocale(locale);
    setlocale(LC_ALL, locale); // Restore the CRT.
    std::wcout.imbue(lollocale); // Now set the std::wcout to have the locale that we got from the CRT.
    std::wcout << L"¡Hola!";
    std::cin.get();
    return 0;
}

I just tested it, and it displays the string here absolutely fine.

Answer 2

Unicode Hello World in Chinese

Here is a Hello World in Chinese. Actually it is just "Hello". I tested this on Windows 10, but I think it might work since Windows Vista. Before Windows Vista it will be hard, if you want a programmatic solution, instead of configuring the console / registry etc. Maybe have a look here if you really need to do this on Windows 7: Change console Font Windows 7

I dont want to claim this is the only solution, but this is what worked for me.

Outline

1. Unicode project setup
2. Set the console codepage to unicode
3. Find and use a font that supports the characters you want to display
4. Use the locale of the language you want to display
5. Use the wide character output i.e. std::wcout

1 Project Setup

I am using Visual Studio 2017 CE. I created a blank console app. The default settings are alright. But if you experience problems or you use a different ide you might want to check these:

In your project properties find configuration properties -> General -> Project Defaults -> Character Set. It should be "Use Unicode Character Set" not "Multi-Byte". This will define _UNICODE and UNICODE preprocessor macros for you.

int wmain(int argc, wchar_t* argv[])

Also I think we should use wmain function instead of main. They both work, but in a unicode environment wmain may be more convenient.

Also my source files are UTF-16-LE encoded, which seems to be the default in Visual Studio 2017.

2. Console Codepage

This is quite obvious. We need the unicode codepage in the console. If you want to check your default codepage, just open a console and type chcp withou any arguments. We have to change it to 65001, which is the UTF-8 codepage. Windows Codepage Identifiers There is a preprocessor macro for that codepage: CP_UTF8. I needed to set both, the input and output codepage. When I omitted either one, the output was incorrect.

SetConsoleOutputCP(CP_UTF8);
SetConsoleCP(CP_UTF8);

You might also want to check the boolean return values of those functions.

3. Choose a Font

Until yet I didnt find a console font that supports every character. So I had to choose one. If you want to output characters which are partly only available in one font and partly in another font, then I believe it is impossible to find a solution. Only maybe if there is a font out there that supports every character. But also I didnt look into how to install a font.

I think it is not possible to use two different fonts in the same console window at the same time.

How to find a compatible font? Open your console, go to the properties of the console window by clicking on the icon in the upper left of the window. Go to the fonts tab and choose a font and click ok. Then try to enter your characters in the console window. Repeat this until you find a font you can work with. Then note down the name of the font.

Also you can change the size of the font in the properties window. If you found a size you are happy with, note down the size values that are displayed in the properties window in the section "selected font". It will show width and height in pixels.

To actually set the font programmatically you use:

CONSOLE_FONT_INFOEX fontInfo;
// ... configure fontInfo
SetCurrentConsoleFontEx(hConsole, false, &fontInfo);

See my example at the end of this answer for details. Or look it up in the fine manual: SetCurrentConsoleFont. This function only exists since Windows Vista.

4. Set the locale

You will need to set the locale to the locale of the language which characters you want to print.

char* a = setlocale(LC_ALL, "chinese");

The return value is interesting. It will contain a string to describe exactly wich locale was chosen. Just give it a try :-) I tested with chinese and german. More info: setlocale

5. Use wide character output

Not much to say here. If you want to output wide characters, use this for example:

std::wcout << L"你好" << std::endl;

Oh, and dont forget the L prefix for wide characters! And if you type literal unicode characters like this in the source file, the source file must be unicode encoded. Like the default in Visual Studio is UTF-16-LE. Or maybe use notepad++ and set the encoding to UCS-2 LE BOM.

Example

Finally I put it all together as an example:

#include <Windows.h>
#include <iostream>
#include <io.h>
#include <fcntl.h>
#include <locale.h>
#include <wincon.h>

int wmain(int argc, wchar_t* argv[])
{
    SetConsoleTitle(L"My Console Window - 你好");
    HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);

    char* a = setlocale(LC_ALL, "chinese");
    SetConsoleOutputCP(CP_UTF8);
    SetConsoleCP(CP_UTF8);

    CONSOLE_FONT_INFOEX fontInfo;
    fontInfo.cbSize = sizeof(fontInfo);
    fontInfo.FontFamily = 54;
    fontInfo.FontWeight = 400;
    fontInfo.nFont = 0;
    const wchar_t myFont[] = L"KaiTi";
    fontInfo.dwFontSize = { 18, 41 };
    std::copy(myFont, myFont + (sizeof(myFont) / sizeof(wchar_t)), fontInfo.FaceName);

    SetCurrentConsoleFontEx(hConsole, false, &fontInfo);

    std::wcout << L"Hello World!" << std::endl;
    std::wcout << L"你好!" << std::endl;
    return 0;
}

Cheers !

Answer 3

There are a few issues with the mswcrt and io streams.

1. Trick _setmode(_fileno(stdout), _O_U16TEXT); working only for MS VC++ not MinGW-GCC. Moreover sometimes it is brings to crashes depending on Windows configuration.
2. SetConsoleCP(65001) for UTF-8. May fail in many multibyte character scenarios, but is is always OK for UTF-16LE
3. You need to restore previews console codepage on application exit.

Windows console supports UNICODE with the ReadConsole and WriteConsole functions in UTF-16LE mode. Background effect - piping in this case will not work. I.e. myapp.exe >> ret.log brings to 0 byte ret.log file. If you are ok with this fact you can try my library as following.

const char* umessage = "Hello!\nПривет!\nПривіт!\nΧαιρετίσματα!\nHelló!\nHallå!\n";

...
#include <console.hpp>
#include <ios>
...

std::ostream& cout = io::console::out_stream();
cout << umessage
<< 1234567890ull << '\n'
<< 123456.78e+09 << '\n'
<< 12356.789e+10L << '\n'
<< std::hex << 0xCAFEBABE
<< std::endl;

Library will auto-convert your UTF-8 into UTF-16LE and write it into console using WriteConsole. As well as there are error and input streams. Another library benefit - colors.

Link on example app: https://github.com/incoder1/IO/tree/master/examples/iostreams

The library homepage: https://github.com/incoder1/IO

Screenshot:

Answer 4

SetConsoleCP() and chcp does not the same!

Take this program snippet:

SetConsoleCP(65001)  // 65001 = UTF-8
static const char s[]="tränenüberströmt™\n";
DWORD slen=lstrlen(s);
WriteConsoleA(GetStdHandle(STD_OUTPUT_HANDLE),s,slen,&slen,NULL);

The source code must be saved as UTF-8 without BOM (Byte Order Mark; Signature). Then, the Microsoft compiler cl.exe takes the UTF-8 strings as-is.
If this code is saved with BOM, cl.exe transcodes the string to ANSI (i.e. CP1252), which doesn't match to CP65001 (= UTF-8).

Change the display font to Lucidia Console, otherwise, UTF-8 output will not work at all.

• Type: chcp
• Answer: 850
• Type: test.exe
• Answer: tr├ñnen├╝berstr├ÂmtÔäó
• Type: chcp
• Answer: 65001 - This setting has changed by SetConsoleCP() but with no useful effect.
• Type: chcp 65001
• Type: test.exe
• Answer: tränenüberströmt™ - All OK now.

Tested with: German Windows XP SP3

Answer 5

Correctly displaying Western European characters in the windows console

Long story short:

1. use chcp to find which codepage works for you. In my case it was chcp 28591 for Western Europe.
2. optionally make it the default: REG ADD HKCU\Console /v CodePage /t REG_DWORD /d 28591

History of the discovery

I had a similar problem, with Java. It is just cosmetic, since it involves log lines sent to the console; but it is still annoying.

The output from our Java application is supposed to be in UTF-8 and it displays correctly in eclipse's console. But in windows console, it just shows the ASCII box-drawing characters: Inicializaci├│n and artículos instead of Inicialización and artículos.

I stumbled upon a related question and mixed some of the answers to get to the solution that worked for me. The solution is changing the codepage used by the console and using a font that supports UNICODE (like consolas or lucida console). The font you can select in the system menu of the Windows cosole:

1. Start a console by any one of
   • Win + R then type cmd and hit the Return key.
   • Hit the Win key and type cmd followed by the return key.
2. Open the system menu by any one of
   • click the upper left corner icon
   • Hit the Alt + Space key combination
3. then select "Default" to change the behavior of all subsequent console windows
4. click the "Font" tab
5. Select Consolas or Lucida console
6. Click OK

Regarding the codepage, for a one-off case, you can get it done with the command chcp and then you have to investigate which codepage is correct for your set of characters. Several answers suggested UTF-8 codepage, which is 65001, but that codepage didn't work for my Spanish characters.

Another answer suggested a batch script to interactively selecting the codepage you wanted from a list. There I found the codepage for ISO-8859-1 I needed: 28591. So you could execute

chcp 28591

before each execution of your application. You might check which code page is right for you in the Code Page Identifiers MSDN page.

Yet another answer indicated how to persist the selected codepage as the default for your windows console. It involves changing the registry, so consider yourself warned that you might brick your machine by using this solution.

REG ADD HKCU\Console /v CodePage /t REG_DWORD /d 28591

This creates the CodePage value with the 28591 data inside the HKCU\Console registry key. And that did work for me.

Please note that HKCU ("HKEY_CURRENT_USER") is only for the current user. If you want to change it for all users in that computer, you'll need to use the regedit utility and find/create the corresponding Console key (probably you'll have to create a Console key inside HKEY_USERS\.DEFAULT)

Answer 6

I have verified a solution here using Visual Studio 2010. Via this MSDN article and MSDN blog post. The trick is an obscure call to _setmode(..., _O_U16TEXT).

Solution:

#include <iostream>
#include <io.h>
#include <fcntl.h>

int wmain(int argc, wchar_t* argv[])
{
    _setmode(_fileno(stdout), _O_U16TEXT);
    std::wcout << L"Testing unicode -- English -- Ελληνικά -- Español." << std::endl;
}

Screenshot: