Why does UTF-8 text sort in different order betwee

Question

I have a text file with lines of UTF-8 encoded text:

mac-os-x$ cat unsorted.txt
ウ
foo
チ
'foo'
津

In case it helps to reproduce the problem, here is a checksum and a dump of the exact bytes in the file, as well as how you could generate the file yourself (on Linux, use base64 -d instead of -D):

mac-os-x$ shasum unsorted.txt
a6d0b708d3e0cafb0c6e1af7450e9243da8cb078  unsorted.txt

mac-os-x$ perl -ne 'print join(" ", map { sprintf "%02x", ord } split //), "\n"' unsorted.txt
e3 82 a6 0a
66 6f 6f 0a
e3 83 81 0a
27 66 6f 6f 27 0a
e6 b4 a5 0a

mac-os-x$ echo 44KmCmZvbwrjg4EKJ2ZvbycK5rSlCg== | base64 -D > unsorted.txt

When I sort this input file on Mac OS X (regardless of whether I use GNU sort 5.93 which Mac OS X Yosemite ships with, or with a Homebrew installed GNU sort version 8.23), I get this sorted result:

mac-os-x$ env -i LANG=en_US.utf-8 LC_ALL=en_US.utf-8 /usr/bin/sort unsorted.txt
'foo'
foo
ウ
チ
津

mac-os-x$ echo `sw_vers -productName` `sw_vers -productVersion`
Mac OS X 10.10.1

mac-os-x$ /usr/bin/sort --version | head -1
sort (GNU coreutils) 5.93

When I sort the same file, with the same locale settings, on Linux (I tested on both Centos 5.5 and CentOS 6.5), I get a different result:

linux-centos-6.5$ env -i LANG=en_US.utf-8 LC_ALL=en_US.utf-8 /bin/sort unsorted.txt
ウ
チ
foo
'foo'
津

linux-centos-6.5$ cat /etc/redhat-release
CentOS release 6.5 (Final)

linux-centos-6.5$ /bin/sort --version | head -1
sort (GNU coreutils) 8.4

Note the different locations of the Japanese kana vs. the English, and the different sort order between two lines that differ only by the single quotes.

To add another variant to the mix, I notice that on a very old FreeBSD 6 box I have, I get the same sort order as OS X:

freebsd-6.0$ env -i LANG=en_US.utf-8 LC_ALL=en_US.utf-8 /usr/bin/sort unsorted.txt
'foo'
foo
ウ
チ
津

freebsd-6.0$ uname -rs
FreeBSD 6.0-RELEASE

freebsd-6.0$ sort --version | head -1
sort (GNU coreutils) 5.3.0-20040812-FreeBSD

I expected the sort order to be the same in each case, given that all cases are using GNU sort, all with the same locale settings. I tried explictly setting LC_COLLATE separately, and tried using LC_COLLATE=C to force a sort by byte order, but that did not change any results.

Why does my example input file sort differently across OS X and Linux? And how could I force both systems to produce identically sorted text (I don't care which variant, as long as it is consistent between the two)?

Answer 1

As it seems - your linux sort is not preserving proper UTF-8 order.

Hex UTF-8 representations of your unsorted.txt (first letters) would be:

ウ - 30A6

foo - 0066

チ - 30C1

'foo' - 0027

津 - 6D25

taken from http://www.ltg.ed.ac.uk/~richard/utf-8.cgi?input=%E3%82%A6&mode=char

So proper sorting according to unicode collation (http://www.unicode.org/Public/UCA/latest/allkeys.txt) would be:

'foo' - line 487

foo - line 8966

ウ - line 20875

チ - line 21004

津 - not in file

So, to answer your question, your linux machine is providing wrong collation tables to sort function. Unfortunately, i can't tell what is possible reason for that.

PS: There's similar question to yours here.

EDIT

As @ninjalj noticed, glibc doesn't use UCA, but ISO-14651 instead. This bug report suggest migration to UCA. Unfortunately, it's still not resolved.

Also, it could be somehow connected with question about ls case insensivity on MacOSX. Some people even suggest that it has something to do with HFS filesystem.