Twisted LoopingCall with blocking function

Question

I have an application which needs to poll a database for possible configuration changes. The application is a simple xmlrpc server using Twisted. I have experimented using Twisted's LoopingCall to perform the polling, but because LoopingCall runs on the main thread, the call to the db is blocking. So if the db call is slow for some reason, requests to the xmlrpc server have to wait. So I tried running the LoopingCall in a thread and couldn't really get it to work. My question is, should I run it in a thread? If so, how?

from twisted.web import xmlrpc, server
from twisted.internet.threads import deferToThread
from twisted.internet import reactor, task
import platform
from time import sleep

r = reactor

class Agent(xmlrpc.XMLRPC):
    self.routine()
    xmlrpc.XMLRPC.__init__(self)

    def xmlrpc_echo(self, x):
        """
        Return arg as a simple test that the server is running
        """
        return x

    def register(self):
        """
        Register Agent with db and pick up config
        """
        sleep(3)  # simulate slow db call
        print 'registered with db'

    def routine(self):
        looping_register = task.LoopingCall(self.register)
        looping_register.start(7.0, True)

if __name__ == '__main__':
    r.listenTCP(7081, server.Site(Agent()))
    print 'Agent is running on "%s"' % platform.node()
    r.run()

Answer 1

You should use twisted.enterprise.adbapi module. It will give you nonblocking api for all DBAPI 2.0 compatible clients by running them in thread pool and returning standard Deferred to you:

from twisted.enterprise import adbapi


dbpool = adbapi.ConnectionPool('psycopg2', 'mydb', 'andrew', 'password') # replace psycopg2 with your db client name.

def getAge(user):
    return dbpool.runQuery('SELECT age FROM users WHERE name = ?', user)

def printResult(l):
    if l:
        print l[0][0], "years old"
    else:
        print "No such user"

getAge("joe").addCallback(printResult)

For more information and examples please visit official documentation: https://twistedmatrix.com/documents/14.0.0/core/howto/rdbms.html