LESSCHARSET=utf-8 less doesn't seem to work

Question

I'm trying to view a UTF-8 text file/stream in less, and even if I invoke it like this:

cat file | LESSCHARSET=utf-8 less

the non-ASCII compatible UTF-8 characters don't display correctly. Instead, their hex values appear highlighted in brackets, e.g. <F4>.

The reading the same text in vim with UTF-8 encoding poses no problems. So I'm thinking something is wrong with the way I'm invoking less.

My locale output is the following

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

My less version is the one installed by XCode on OSX Leopard:

$ less --version | sed 's/^/    /'
less 394
Copyright (C) 1984-2005 Mark Nudelman

less comes with NO WARRANTY, to the extent permitted by law.
For information about the terms of redistribution, 
see the file named README in the less distribution.
Homepage: http://www.greenwoodsoftware.com/less

locale -a | grep US | sed 's/^/ /' outputs the following:

en_AU.US-ASCII
en_CA.US-ASCII
en_GB.US-ASCII
en_NZ.US-ASCII
en_US
en_US.ISO8859-1
en_US.ISO8859-15
en_US.US-ASCII
en_US.UTF-8

Answer 1

Try the command file file.txt.  If, for example, the output is "ISO-8859 English text" then change the encoding of the file from ISO-8859 to UTF-8 via the command iconv -f ISO-8859-1 -t UTF-8 -o testfile.txt file.txt.  If less testfile.txt displays correctly, finish with mv testfile.txt file.txt.

Answer 2

On Mac OS a charset have to be uppercase:

bash-4.4$ less --version
less 458 (POSIX regular expressions)
Copyright (C) 1984-2012 Mark Nudelman

bash-4.4$ LESSCHARSET=cp1251 less
invalid charset name

bash-4.4$ LESSCHARSET=CP1251 less
Missing filename ("less --help" for help)

Here I found list of charsets:

{ "ascii",          NULL,       "8bcccbcc18b95.b" },
{ "utf-8",          &utf_mode,  "8bcccbcc18b95.b126.bb" },
{ "iso8859",        NULL,       "8bcccbcc18b95.33b." },
{ "latin3",         NULL,       "8bcccbcc18b95.33b5.b8.b15.b4.b12.b18.b12.b." },
{ "arabic",         NULL,       "8bcccbcc18b95.33b.3b.7b2.13b.3b.b26.5b19.b" },
{ "greek",          NULL,       "8bcccbcc18b95.33b4.2b4.b3.b35.b44.b" },
{ "greek2005",      NULL,       "8bcccbcc18b95.33b14.b35.b44.b" },
{ "hebrew",         NULL,       "8bcccbcc18b95.33b.b29.32b28.2b2.b" },
{ "koi8-r",         NULL,       "8bcccbcc18b95.b." },
{ "KOI8-T",         NULL,       "8bcccbcc18b95.b8.b6.b8.b.b.5b7.3b4.b4.b3.b.b.3b." },
{ "georgianps",     NULL,       "8bcccbcc18b95.3b11.4b12.2b." },
{ "tcvn",           NULL,       "b..b...bcccbccbbb7.8b95.b48.5b." },
{ "TIS-620",        NULL,       "8bcccbcc18b95.b.4b.11b7.8b." },
{ "next",           NULL,       "8bcccbcc18b95.bb125.bb" },
{ "dos",            NULL,       "8bcccbcc12bc5b95.b." },
{ "windows-1251",   NULL,       "8bcccbcc12bc5b95.b24.b." },
{ "windows-1252",   NULL,       "8bcccbcc12bc5b95.b.b11.b.2b12.b." },
{ "windows-1255",   NULL,       "8bcccbcc12bc5b95.b.b8.b.5b9.b.4b." },
{ "ebcdic",         NULL,       "5bc6bcc7bcc41b.9b7.9b5.b..8b6.10b6.b9.7b9.8b8.17b3.3b9.7b9.8b8.6b10.b.b.b." },
{ "IBM-1047",       NULL,       "4cbcbc3b9cbccbccbb4c6bcc5b3cbbc4bc4bccbc191.b" },
{ NULL, NULL, NULL }

and their aliases:

{ "UTF-8",          "utf-8" },
{ "ANSI_X3.4-1968", "ascii" },
{ "US-ASCII",       "ascii" },
{ "latin1",         "iso8859" },
{ "ISO-8859-1",     "iso8859" },
{ "latin9",         "iso8859" },
{ "ISO-8859-15",    "iso8859" },
{ "latin2",         "iso8859" },
{ "ISO-8859-2",     "iso8859" },
{ "ISO-8859-3",     "latin3" },
{ "latin4",         "iso8859" },
{ "ISO-8859-4",     "iso8859" },
{ "cyrillic",       "iso8859" },
{ "ISO-8859-5",     "iso8859" },
{ "ISO-8859-6",     "arabic" },
{ "ISO-8859-7",     "greek" },
{ "IBM9005",        "greek2005" },
{ "ISO-8859-8",     "hebrew" },
{ "latin5",         "iso8859" },
{ "ISO-8859-9",     "iso8859" },
{ "latin6",         "iso8859" },
{ "ISO-8859-10",    "iso8859" },
{ "latin7",         "iso8859" },
{ "ISO-8859-13",    "iso8859" },
{ "latin8",         "iso8859" },
{ "ISO-8859-14",    "iso8859" },
{ "latin10",        "iso8859" },
{ "ISO-8859-16",    "iso8859" },
{ "IBM437",         "dos" },
{ "EBCDIC-US",      "ebcdic" },
{ "IBM1047",        "IBM-1047" },
{ "KOI8-R",         "koi8-r" },
{ "KOI8-U",         "koi8-r" },
{ "GEORGIAN-PS",    "georgianps" },
{ "TCVN5712-1",     "tcvn" },
{ "NEXTSTEP",       "next" },
{ "windows",        "windows-1252" }, /* backward compatibility */
{ "CP1251",         "windows-1251" },
{ "CP1252",         "windows-1252" },
{ "CP1255",         "windows-1255" },
{ NULL, NULL }

Answer 3

1. What does the locale command output? Is it a UTF-8 locale?

2. Are you sure your terminal is set to display UTF-8? Does echo -e '\xe2\x82\xac' produce the € (euro) sign?

3. Is the locale that you have set even installed on the system? Is it present in the list that locale -a outputs?

4. What version of less are you using? (Run less --version to find out.) Really, really old versions did not even support LESSCHARSET. This is less likely to be the case, because I have a Debian "sarge" system with less version 382, and it does not even need LESSCHARSET if the locale is set correctly.

Answer 4

My guess is that your file isn't UTF8 but rather ISO8859. (Is the <F4> character supposed to be a 'ô'?)

Start an xterm with LANG=en_US.ISO-8859-1 xterm. Then verify the locale (the output of locale should be something like en_US.ISO-8859-1). Then use less to view the file. Does it display correctly?

Note that it isn't enough to just use LESSCHARSET=iso8859 without starting a new terminal. LESSCHARSET tells less to think that the terminal can interpret iso8859, but your terminal probably displays UTF8, since the euro sign displays correctly. But as \xf4 isn't a valid utf8 character, the terminal will probably show something like '�'.