tweepy stream to sqlite database - invalid synatx

问题:

The code below is streaming the twitter public timeline for a variable which output any tweets to the console. I'd like the save the same variables (status.text, status.author.screen_name, status.created_at, status.source) into an sqlite database. I'm getting an syntax error when my script sees a tweet and nothing is written to the sqlite database.

the error:

$ python stream-v5.py @lunchboxhq
Filtering the public timeline for "@lunchboxhq"RT @LunchboxHQ: test 2   LunchboxHQ  2012-02-29 18:03:42 Echofon
Encountered Exception: near "?": syntax error

the code:

import sys
import tweepy
import webbrowser
import sqlite3 as lite

# Query terms

Q = sys.argv[1:]

sqlite3file='/var/www/twitter.lbox.com/html/stream5_log.sqlite'

CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_TOKEN = ''
ACCESS_TOKEN_SECRET = ''

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

con = lite.connect(sqlite3file)
cur = con.cursor()
cur.execute("CREATE TABLE TWEETS(txt text, author text, created int, source text)")

class CustomStreamListener(tweepy.StreamListener):

    def on_status(self, status):

        try:
            print "%s\t%s\t%s\t%s" % (status.text, 
                                      status.author.screen_name, 
                                      status.created_at, 
                                      status.source,)

            cur.executemany("INSERT INTO TWEETS(?, ?, ?)", (status.text, 
                                                            status.author.screen_name, 
                                                            status.created_at, 
                                                            status.source))

        except Exception, e:
            print >> sys.stderr, 'Encountered Exception:', e
            pass

    def on_error(self, status_code):
        print >> sys.stderr, 'Encountered error with status code:', status_code
        return True # Don't kill the stream

    def on_timeout(self):
        print >> sys.stderr, 'Timeout...'
        return True # Don't kill the stream

streaming_api = tweepy.streaming.Stream(auth, CustomStreamListener(), timeout=60)

print >> sys.stderr, 'Filtering the public timeline for "%s"' % (' '.join(sys.argv[1:]),)

streaming_api.filter(follow=None, track=Q)

回答1:

You are missing a closing parenthesis on the last line of the following code (lines 34–37 from what you posted):

            cur.executemany("INSERT INTO TWEETS(?, ?, ?)", (status.text, 
                                                        status.author.screen_name, 
                                                        status.created_at, 
                                                        status.source)

Just add a parenthesis to close the method call immediately after your tuple parameter.

回答2:

import sqlite3 as lite
con = lite.connect('test.db')
cur = con.cursor()   

cur.execute("CREATE TABLE TWEETS(txt text, author text, created int, source text)")

then later:

cur.executemany("INSERT INTO TWEETS(?, ?, ?, ?)", (status.text, 
                                      status.author.screen_name, 
                                      status.created_at, 
                                      status.source))

回答3:

Full disclosure: still new to this stuff. However, I got your code working by changing it to:

cur.execute("INSERT INTO TWEETS VALUES(?,?,?,?)", (status.text, status.author.screen_name, status.created_at, status.source))
con.commit()

It seems to me that you're reading in one status at a time. The executemany method would be for when you have more than one status. For example:

(['sometext', 'bob','2013-02-01','Twitter for Android'], ['someothertext', 'helga', '2013-01-31', 'MacSomething'])

I'm definitely not a wizard and am not sure what sort of impact the commit() has on every entry... I'm guessing the performance is terrible, but it works for a single term in the query.

Thanks for posting your code, I finally learned how to do streaming.

回答4:

I'm quite new to tweepy . But these are the modifications that worked for me . You need to add VALUES after INSERT INTO TWEETS . Also , don't forget to commit the changes . This is the link I referred to : related post

     cur.execute("INSERT INTO TWEETS VALUES(?, ?, ?, ?)", (status.text, 
                                                        status.author.screen_name, 
                                                        status.created_at, 
                                                        status.source))

     con.commit()