Flask app broken after bson update in Heroku

Question

I have a Flask app that uses mongoengine and running on Heroku, init I use the bson package and after I updateded it from 0.5.6 to 0.5.7 I started getting the following error message:

[2018-11-23 05:56:43 +0000] [39] [INFO] Worker exiting (pid: 39)
[2018-11-23 05:56:43 +0000] [40] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/app/slots_tracker_server/__init__.py", line 5, in <module>
    from mongoengine import connect
  File "/app/.heroku/python/lib/python3.6/site-packages/mongoengine/__init__.py", line 2, in <module>
    from mongoengine import connection
  File "/app/.heroku/python/lib/python3.6/site-packages/mongoengine/connection.py", line 1, in <module>
    from pymongo import MongoClient, ReadPreference, uri_parser
  File "/app/.heroku/python/lib/python3.6/site-packages/pymongo/__init__.py", line 77, in <module>
    from pymongo.collection import ReturnDocument
  File "/app/.heroku/python/lib/python3.6/site-packages/pymongo/collection.py", line 20, in <module>
    from bson.code import Code
  File "/app/.heroku/python/lib/python3.6/site-packages/bson/code.py", line 18, in <module>
    from bson.py3compat import abc, string_type, PY3, text_type
ImportError: cannot import name 'abc'

This error appears when I restart my server, after that I don't see errors in the log but I can't access the app. I'm running the same configuration in Travis-CI but there I don't see any issues. Any idea on how I can fix it?

This is the output of pip freeze in my server:

blinker==1.4
bson==0.5.7
CacheControl==0.12.5
cachetools==3.0.0
certifi==2018.10.15
chardet==3.0.4
Click==7.0
dateutils==0.6.6
firebase-admin==2.13.0
Flask==1.0.2
Flask-Cors==3.0.7
google-api-core==1.5.2
google-auth==1.6.0
google-cloud-core==0.28.1
google-cloud-firestore==0.30.0
google-cloud-storage==1.13.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.5
grpcio==1.16.1
gspread==3.0.1
gunicorn==19.9.0
httplib2==0.11.3
idna==2.7
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
mongoengine==0.16.1
msgpack==0.5.6
numpy==1.15.4
oauth2client==4.1.3
pandas==0.23.4
protobuf==3.6.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pyfcm==1.4.5
pymongo==3.7.2
python-dateutil==2.7.5
python-http-client==3.1.0
pytz==2018.7
raven==6.9.0
requests==2.20.1
requests-toolbelt==0.8.0
rsa==4.0
sendgrid==5.6.0
six==1.11.0
urllib3==1.24.1
Werkzeug==0.14.1

Note that pymongo>=2.7.1 is installed as a dependency of mongoengine

Update: After removing bson and adding pymongo==3.7.2 to the requiements file, I got the following error:

Traceback (most recent call last):
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
     worker.init_process()
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
     self.load_wsgi()
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
     self.wsgi = self.app.wsgi()
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
     self.callable = self.load()
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
     return self.load_wsgiapp()
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
     return util.import_app(self.app_uri)
   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
     __import__(module)
   File "/app/slots_tracker_server/__init__.py", line 5, in <module>
     from mongoengine import connect
   File "/app/.heroku/python/lib/python3.6/site-packages/mongoengine/__init__.py", line 2, in <module>
     from mongoengine import connection
   File "/app/.heroku/python/lib/python3.6/site-packages/mongoengine/connection.py", line 1, in <module>
     from pymongo import MongoClient, ReadPreference, uri_parser
   File "/app/.heroku/python/lib/python3.6/site-packages/pymongo/__init__.py", line 77, in <module>
     from pymongo.collection import ReturnDocument
   File "/app/.heroku/python/lib/python3.6/site-packages/pymongo/collection.py", line 20, in <module>
     from bson.code import Code
   File "/app/.heroku/python/lib/python3.6/site-packages/bson/code.py", line 18, in <module>
     from bson.py3compat import abc, string_type, PY3, text_type
 ModuleNotFoundError: No module named 'bson.py3compat'

Answer 1

pymongo>=2.7.1 directive overwrites your bson installation

from https://github.com/mongodb/mongo-python-driver

Do not install the "bson" package from pypi. PyMongo comes with its own bson package; doing "easy_install bson" installs a third-party package that is incompatible with PyMongo.

assuming you really want to use the 3rd party bson, you'll need to

1. pip uninstall bson (removes the overwritten bson that landed via pymongo package)
2. pip install bson (reinstalls the py-bson package)

or you could patch the pymongo install to remove the collections.abc line (evidently you don't really need it?), but messing with production packaged code is questionable.

pymongo module https://github.com/mongodb/mongo-python-driver/blob/master/bson/py3compat.py#L22-L25

py-bson module you are trying to use. https://github.com/py-bson/bson/blob/master/bson/py3compat.py#L22-L24