How should I use g++'s -finput-charset compile

Question

I'm trying to compile a UTF-16BE C++ source file in g++ with -finput-charset compiler option but I'm always getting a bunch of errors. More details follow.

My environment(in CentOS Linux):

• g++: 4.1.2
• iconv: 2.5
• Linux language(in Terminal): LANG="en_US.UTF-8"

My sample source file(stored in UTF-16BE encoding):

// main.cpp:

#include <iostream>

int main()
{
    std::cout << "Hello, UTF-16" << std::endl;
    return 0;
}

My steps:

• I read the manual of g++ about the -finput-charset option. The g++ manual says:

-finput-charset=charset Set the input character set, used for translation from the character set of the input file to the source character set used by GCC. If the locale does not specify, or GCC cannot get this information from the locale, the default is UTF-8. This can be overridden by either the locale or this command line option. Currently the command line option takes precedence if there’s a conflict. charset can be any encoding supported by the system’s "iconv" library routine.

• Thus I entered the command as follows:

g++ -finput-charset=UTF-16BE main.cpp

and I got these errors:

In file included from main.cpp:1:

/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1: error: stray ‘\342’ in program

/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1: error: stray ‘\274’ in program

...(repeatedly, A LOT, around 4000+)...

/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1: error: stray ‘\257’ in program

main.cpp: In function ‘int main()’:

main.cpp:5: error: ‘cout’ is not a member of ‘std’

main.cpp:5: error: ‘endl’ is not a member of ‘std’

• The manual text suggests that the charset can be any encoding supported by 'iconv' routine, thus I guessed the compilation errors might be caused by my iconv library. I then tested the iconv:

iconv --from-code=UTF-16BE --to-code=UTF-8 --output=main_utf8.cpp main.cpp

A "main_utf8.cpp" file is generated as expected. I then tried to compile it:

g++ -finput-charset=UTF-8 main_utf8.cpp

Note that I specified the input-charset explicitly to see if I did anything wrong, but this time a "a.out" was generated without any errors. When I ran it, it could produce the correct output.

Finally...

I couldn't figure out where I did wrong. I searched in the web trying to find out some examples for this compiler option but I couldn't.

Please advise! Thanks!

Further edits:

Thanks, guys! Your replies are quick! Some updates:

1. When I said "UTF-16" I meant "UTF-16 + BOM". In fact I used UTF-16BE. I have updated the text above.
2. Some answers say the errors are caused by the non-UTF-16 header files. Here are my thoughts if this is the case: We'll always include some standard header files when writing a C/C++ project, right? Such as stdio.h or iostream. If the G++ compiler only deals with the encoding of the source files created by us but never with the source files in the standard library, then what does this -finput-charset option exist for??

Final edit:

At last, my solution is like this:

1. At the beginning, I changed the encoding of my source files to GB2312, as "Mr Lister" said below. This worked fine for a while, but later I found it not suitable for my situation because most of the other parts in the system still use UTF-8 for communication and interfaces, thus I must convert the encoding in many places... Not only an overhead of my work, it may also result in some performance decrease in my program.
2. Later I tried to convert all my source files to UTF-8 + BOM. In this way, Visual Studio in Windows could compile them happily but GCC in Linux would complain. I then wrote a shell script to remove the BOM, and before I want to compile my code with GCC, I run this script first.
3. Luckily, I don't have to build the code in Linux manually because TeamCity the continuous integration tool is used in my project to generate the build automatically. I could change the build steps in TeamCity to help me run this script before the daily build starts.
4. With this UTF-8 + BOM + script method, I decide not to edit my source code in Linux, because if I want to do so, I must make sure my code could build successfully before I commit it, which means I must run the script to remove the BOM before I build the code, which means SVN would report EVERY file is modified(BOM removed) thus make it very easy to mistakenly commit a wrong file. To solve this problem, I wrote another shell script to add the BOM back to the source files. Though I still don't edit my code very often in Linux, but when I really need to, I don't have to face the terribly long change list in the commit dialog.

Answer 1

UTF-16 is not an encoding for bytes. It's an encoding where your basic storage unit is 16 bits large.

When you want to store UTF-16 in a byte sequence you have to choose between UTF-16BE and UTF-16LE.

Answer 2

The error message says the problem is in the include files, so I presume what happens is that the include files are normal UTF-8, but the compiler wants to treat them as UTF-16 because of the compiler switch.

So I'm afraid the solution is to always convert the source to UTF-8 first; perhaps in the makefile. Or to find a solution that doesn't contain include files in other encodings...

Edit: Maybe a GB encoding would work, if and only if none of the system source files contain any non-ASCII characters. Then you could tell the compiler they were GB encoded without problem.

Answer 3

This does not work because the compiler will also try to read the header files as UTF-16, which they are not.

Answer 4

Encoding Blues

You cannot use UTF-16 for source code files; because the header you are including, <iostream>, is not UTF-16-encoded. As #include includes the files verbatim, this means that you suddenly have an UTF-16-encoded file with a large chunk (approximately 4k, apparently) of invalid data.

There is almost no good reason to ever use UTF-16 for anything, so this is just as well.

Edit: Regarding problems with encoding support: The OSes themselves are not responsible for providing encoding support, this comes down to the compilers used.

g++ on Windows supports absolutely all of the same encodings as g++ on Linux, because it's the same program, unless whatever version of g++ you are using on Windows relies on a deeply broken iconv library.

Inspect your toolchain and ensure that all your tools are in working order.

As an alternative; don't use Chinese in the source files, but write them in English, using English-language literals, or simple TOKEN_STYLE_PLACEHOLDERs, using l10n and i18n to replace these in the running executable.

Threedit: -finput-charset is almost certainly a holdover from the days of codepages and other nonsense of the kind; however; an ISO-8859-n file will almost always be compatible with UTF-8 standard headers, however, see the reedit below.

Reedit: For next time; remember a simple mantra: "N'DUUH!"; "Never Don't Use UTF-8!"

---

I18N

A common solution to this kind of problem is to remove the problem entirely, by way of, for instance, gettext.

When using gettext, you usually end up with a function loc(char *) that abstracts away most of the translation tool specific code. So, instead of

#include <iostream>

int main () {
  std::cout << "瓜田李下" << std::endl;
}

you would have

#include <iostream>

#include "translation.h"

int main () {
  std::cout << loc("DEEPER_MEANING") << std::endl;
}

and, in zh.po:

msgid DEEPER_MEANING
msgstr "瓜田李下"

Of course, you could also then have a en.po:

msgid DEEPER_MEANING
msgstr "Still waters run deep"

This can be expanded upon, and the gettext package has tools for expansion of strings with variables and such, or you could use printf, to account for different grammars.

---

The Third Option

Instead of having to deal with multiple compilers with different requirements for file encodings, file endings, byte order marks, and other problems of the kind; it is possible to cross-compile using MinGW or similar tools.

This option requires some setup, but may very well reduce future overhead and headaches.