Django celery running only two tasks at once?

Question

I have a celery task like this:

@celery.task
def file_transfer(password, source12, destination):
    result = subprocess.Popen(['sshpass', '-p', password, 'rsync', '-avz', source12, destination], 
                                    stderr=subprocess.PIPE, stdout=subprocess.PIPE).communicate()[0]                             
    return result        

I have called in a Djagno view.

User can select more than one file to copy to the destination. For example if the user selects, 4 files at once, celery accept only 2 tasks. What's wrong?

Answer 1

Have you checked the concurrency setting of your worker ?

If for example you have only one worker running on a two-core machine, concurrency by default will be 2. Which means only two tasks can be executed at once.

You can change this setting from the worker command-line with the switch:

 -c N

where N is number of parallel tasks