In PHP, how do I deal with the difference in encod

Question

I am creating a very simple file search, where the search database is a text file with one file name per line. The database is built with PHP, and matches are found by grepping the file (also with PHP).

This works great in Linux, but not on Mac when non-ascii characters are used. It looks like names are encoded differently on HFS+ (MacOSX) than on e.g. ext3 (Linux). Here's a test.php:

<?php
$mystring = "abcóüÚdefå";
file_put_contents($mystring, "");
$h = dir('.');
$h->read(); // "."
$h->read(); // ".."
$filename = $h->read();

print "string: $mystring and filename: $filename are ";

if ($mystring == $filename) print "equal\n";
else print "different\n";

When run MacOSX:

$ php test.php
string: abcóüÚdefå and filename: abcóüÚdefå are different
$ php test.php |cat -evt
string: abcóü?M-^Zdefå$ and filename: abco?M-^Au?M-^HU?M-^Adefa?M-^J are different$

When run on Linux (or on a nfs-mounted ext3 filesystem on MacOSX):

$ php test.php
string: abcóüÚdefå and filename: abcóüÚdefå are equal
$ php test.php |cat -evt
string: abcM-CM-3M-CM-<M-CM-^ZdefM-CM-% and filename: abcM-CM-3M-CM-<M-CM-^ZdefM-CM-% are equal$

Is there a way to make this script return "equal" on both platforms?

Answer 1

MacOSX uses normalization form D (NFD) to encode UTF-8, while most other systems use NFC.

(from unicode.org)

There are several implementations on NFD to NFC conversion. Here I've used the PHP Normalizer class to detect NFD strings and convert them to NFC. It's available in PHP 5.3 or through the PECL Internationalization extension. The following amendment will make the script work:

...
$filename = $h->read();
if (!normalizer_is_normalized($filename)) {
   $filename = normalizer_normalize($filename);
}
...

Answer 2

It seems that Mac OS X/HFS+ is using character combinations instead of single characters. So the ó (U+00F3) is instead encoded as o (U+006F) + ´ (U+CC81, COMBINING ACUTE ACCENT). See also Apple’s Unicode Decomposition Table.

Answer 3

Have you checked that both systems use the same locale?

What encoding is the PHP script using on both systems?

I would also try using strcmp instead of the equals operator. I'm not sure if the equals operator uses strcmp internally, but it's a simple thing to test out in your case.