How to remove multiple UTF-8 BOM sequences before

Question

Using PHP5 (cgi) to output template files from the filesystem and having issues spitting out raw HTML.

private function fetch($name) {
    $path = $this->j->config['template_path'] . $name . '.html';
    if (!file_exists($path)) {
        dbgerror('Could not find the template "' . $name . '" in ' . $path);
    }
    $f = fopen($path, 'r');
    $t = fread($f, filesize($path));
    fclose($f);
    if (substr($t, 0, 3) == b'\xef\xbb\xbf') {
        $t = substr($t, 3);
    }
    return $t;
}

Even though I've added the BOM fix I'm still having problems with Firefox accepting it. You can see a live copy here: http://ircb.in/jisti/ (and the template file I threw at http://ircb.in/jisti/home.html if you want to check it out)

Any idea how to fix this? o_o

Answer 1

Another way to remove the BOM which is Unicode code point U+FEFF

$str = preg_replace('/\x{FEFF}/u', '', $file);

Answer 2

If you are reading some API using file_get_contents and got an inexplicable NULL from json_decode, check the value of json_last_error(): sometimes the value returned from file_get_contents will have an extraneous BOM that is almost invisible when you inspect the string, but will make json_last_error() to return JSON_ERROR_SYNTAX (4).

>>> $json = file_get_contents("http://api-guiaserv.seade.gov.br/v1/orgao/all");
=> "\t{"orgao":[{"Nome":"Tribunal de Justi\u00e7a","ID_Orgao":"59","Condicao":"1"}, ...]}"
>>> json_decode($json);
=> null
>>>

In this case, check the first 3 bytes - echoing them is not very useful because the BOM is invisible on most settings:

>>> substr($json, 0, 3)
=> "  "
>>> substr($json, 0, 3) == pack('H*','EFBBBF');
=> true
>>>

If the line above returns TRUE for you, then a simple test may fix the problem:

>>> json_decode($json[0] == "{" ? $json : substr($json, 3))
=> {#204
     +"orgao": [
       {#203
         +"Nome": "Tribunal de Justiça",
         +"ID_Orgao": "59",
         +"Condicao": "1",
       },
     ],
     ...
   }

Answer 3

This might help. let me know if you care for me to expand my thought process.

<?php
    //
    // labled TESTINGSTRIPZ.php
    //

    define('CHARSET', 'UTF-8');

    $stringy = "\xef\xbb\xbf\"quoted text\" ";
    $str_find_array    = array( "\xef\xbb\xbf");
    $str_replace_array = array(             '');


    $RESULT =
        trim(
            mb_convert_encoding(

                str_replace(
                    $str_find_array,
                    $str_replace_array,
                    strip_tags( $stringy )
                    ),

                'UTF-8',

                mb_detect_encoding(
                    strip_tags($stringy)
                    )

                )
            );

        print("YOUR RESULT IS: " . $RESULT.PHP_EOL);

?>

Result:

terminal$ php TESTINGSTRIPZ.php 
      YOUR RESULT IS: "quoted text" // < with no hidden char.