Corrupted UTF-8 encoding when reading Google feed

2019-07-21 05:52发布

Whenever I try to read a Google alert via PHP using something like:

$feed = file_get_contents("http://www.google.com/alerts/feeds/01445174399729103044/950192755411504138");

Regardless of whether I save the $feed to a file or echo the result to the output, all utf-8 unicode characters ( i.e. those with diacritics) are represented by white space. I have tried - without success - various combinations of:

  • utf8_encode
  • utf8_decode
  • iconv
  • mb_convert_encoding

I think the wrong characters have come from the stream, but I'm lost because if I try this URI in a browser then everything is fine. Can anyone shed some light on the issue?

1条回答
来,给爷笑一个
2楼-- · 2019-07-21 06:56

Sorry, you are absolutely correct - there is something untoward happening! Though it is not what you would first suspect... For reference, given that:

echo mb_detect_encoding($feed); // prints: ASCII

The unicode data is lost before it is even sent by the remote server - it appears that Google is looking at the user-agent string in the request header - which is non-existent using file_get_contents by default without a stream-context.

Because it cannot identify the client making the request it defaults to and forces ASCII encoding. This is presumably a necessary fallback in the event of some kind of cataclysmic cock-up. [citation needed...]

It's not simply enough to name your application however, you need to include a known vendor. I 'm unsure of the full extent of this but I believe most folks include "Mozilla [version]" to work around the issue, for example:

$url = 'http://www.google.com/...';

$feed = file_get_contents($url, false, stream_context_create([
    'http' => [
        'method' => 'GET',
        'header' => 'Accept-Charset: UTF-8' ."\r\n"
                   .'User-Agent: (Mozilla/5.0 compatible) MyFeedReader/1.0'
    ]
]));

file_put_contents('test.txt', $feed); // should now work as expected
查看更多
登录 后发表回答