I have a web scraper that takes forum questions, splits them into individual words and writes it to the text file. The words are stored in a list of tuples. Each tuple contains the word and its frequency. Like so...
[(u'move', 3), (u'exploration', 4), (u'prediction', 21),
(u'find', 5), (u'user', 2), (u'interface', 2), (u'pleasant', 2),
(u'am', 11), (u'puzzled', 2), (u'find', 5), (u'way', 5),
(u'prediction', 21), (u'mode', 2), (u'have', 21),
(u'explored', 2), (u'file', 9), (u'Can', 7), (u'help', 6),
(u'Possible', 1), (u'bug', 2), (u'data', 31), (u'is', 17)
however, some person on the forum used the character \u200b which breaks all my code because that character is no longer a Unicode whitespace.
(u'used\u200b', 1)
Printing it out does not produce an error, but writing to a text file does. I have found that string.strip()
and string.replace()
do not help, so I was wondering how to use the regex library to get rid of that character. I plan on parsing through the entire list of tuples to find it.