importing JSON to mongoDB using pymongo

Question

i am trying to import a JSON file i pull from a URL and send it to mongoDB as is, using the pymongo module.

I have the following code

#!/usr/bin/env python
import sys, urllib2, json, pymongo
from pymongo import MongoClient
myurl = "https://gist.githubusercontent.com/border/775526/raw/b921df18ba00262ab5bba8cadb3c178e1f7748f7/config.json"
response = urllib2.urlopen(myurl)
data = response.read()
connection = MongoClient('mongodb://user:[e-mail]:27017/database')
connection.database_names()
db = connection.database
posts = db.posts
post_id = posts.insert_many(data).inserted_id

upon executing this, i get this error raise TypeError("documents must be a non-empty list") TypeError: documents must be a non-empty list

ideally, i want to just be able to pull the json from the url and update the mongoDB as this json file will be updated every week. Thanks

Answer 1

You need to convert JSON to Python objects, which PyMongo will then convert to BSON for sending to MongoDB. To convert JSON to Python objects use the "bson.json_util" module included with PyMongo:

from bson import json_util
data = json_util.loads(response.read())

The standard Python json.loads() function works, too, but PyMongo's json_util.loads() handles some MongoDB-specific details better.

Answer 2

You can also use the "requests" library:

response = requests.get(myurl)
data = response.json()

and post that data to the database.