Python Reddis Queue ValueError: Functions from the

Question

I'm trying to enqueue a basic job in redis using python-rq, But it throws this error

"ValueError: Functions from the main module cannot be processed by workers"

Here is my program:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

Answer 1

Break the provided code to two files: count_words.py:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

and main.py (where you'll import the required function):

from rq import Connection, Queue
from redis import Redis
from count_words import count_words_at_url # added import!
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

I always separate the tasks from the logic running those tasks to different files. It's just better organization. Also note that you can define a class of tasks and import/schedule tasks from that class instead of the (over-simplified) structure I suggest above. This should get you going.. Also see here to confirm you're not the first to struggle with this example. RQ is great once you get the hang of it.