{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://www.manongdao.com/data/attach/logo/logo.png', '推荐 Evening l夕情丶 的问题《Encoding detection library in python [duplicate]》','https://www.manongdao.com/q-583574.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}
This question already has an answer here:
- How to determine the encoding of text? 8 answers
This is somehow related to my question here.
I process tons of texts (in HTML and XML mainly) fetched via HTTP. I'm looking for a library in python that can do smart encoding detection based on different strategies and convert texts to unicode using best possible character encoding guess.
I found that chardet does auto-detection extremely well. However auto-detecting everything is the problem because it is SLOW and very much against all standards. As per chardet FAQ I don't want to screw the standards.
From the same FAQ here is the list of places where I want to look for encoding:
- charset parameter in HTTP
Content-typeheader. <meta http-equiv="content-type">element in the<head>of a web page for HTML documents.- encoding attribute in the XML prolog for XML documents.
- Auto-detect the character encoding as a last resort.
Basically I want to be able to look in all those place and also deal with conflicting information automatically.
Is there such library out there or do I need to write it myself?
BeautifulSoup's UnicodeDammit, which in turn uses chardet.
chardetby itself is quite useful for the general case (determining text's encoding) but slow as you say.UnicodeDammitadds extra features on top ofchardet, in particular that it can look up the encoding explicitly specified in XML's encoding tags.As for the HTTP
Content-typeheader, I think you need to read that yourself to extract thecharsetparameter, and then pass it toUnicodeDammitin thefromEncodingparameter.As for resolving conflicts, UnicodeDammit will give precedence to explicitly-stated encoding (if the encoding doesn't generate errors). See the docs for full details.
BeautifulSoup (the html parser) incorporates a class called UnicodeDammit that does just that. Have a look and see if you like it.