import error due to bs4 vs BeautifulSoup

Question

I am trying to use beautifulsoup compatible lxml and it is giving me an error:

from lxml.html.soupparser import fromstring
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/lxml/html/soupparser.py", line 7, in <module>
    from BeautifulSoup import \
ImportError: No module named BeautifulSoup

I have bs4 installed. How do I fix this issue?

Answer 1

Try adding :

from bs4 import BeautifulSoup

and make sure you have the correct version of BeautifulSoup for your system installed.

Answer 2

There is now a version of soupparser that works with bs4. It's available here: https://github.com/lxml/lxml/blob/master/src/lxml/html/soupparser.py

Answer 3

The error is caused by soupparser.py trying to import BeautifulSoup version 3 while you have version 4 installed. The module name was changed from BeautifulSoup to bs4 in version 4.

You can trick soupparser.py into importing version 4 by mapping the bs4 module to BeautifulSoup in sys.modules before importing soupparser:

import sys, bs4
sys.modules['BeautifulSoup'] = bs4

from lxml.html.soupparser import fromstring