Django cache.set() causing duplicate key error

Question

My Django site recently started throwing errors from my caching code and I can't figure out why...

I call:

from django.core.cache import cache
cache.set('blogentry', some_value)

And the error thrown by Django is:

TransactionManagementError: This code isn't under transaction management

But looking at the PostgreSQL database logs, it seems to stem from this error:

STATEMENT:  INSERT INTO cache_table (cache_key, value, expires) VALUES (E'blogentry', E'pickled_version_of_some_value', E'2009-07-27 11:10:26')
ERROR:  duplicate key value violates unique constraint "cache_table_pkey"

For the life of me I can't figure out why Django is trying to do an INSERT instead of an UPDATE. Any thoughts?

Answer 1

The code in core/cache/backend/db.py reads in part:

cursor.execute("SELECT cache_key, expires FROM %s WHERE cache_key = %%s" % self._table, [key])
try:
    result = cursor.fetchone()
    if result and (mode == 'set' or
            (mode == 'add' and result[1] < now)):
        cursor.execute("UPDATE %s SET value = %%s, expires = %%s WHERE cache_key = %%s" % self._table, [encoded, str(exp), key])
    else:
        cursor.execute("INSERT INTO %s (cache_key, value, expires) VALUES (%%s, %%s, %%s)" % self._table, [key, encoded, str(exp)])

So I'd say that you are doing the INSERT INTO instead of the UPDATE because result evaluates to false. For some reason, cursor.fetchone() returns 0 rows when there is actually one there.

if you can't break in a debugger here, I'd put trace statements into the source to confirm that this is actually happening.

Answer 2

That's a typical race. It checks if the key you inserted exists; if it doesn't, it does an insert, but someone else can insert the key between the count and the insert. Transactions don't prevent this.

The code appears to expect this and to try to deal with it, but when I looked at the code to handle this case I could see immediately that it was broken. Reported here: http://code.djangoproject.com/ticket/11569

I'd strongly recommend sticking to the memcache backend.

Answer 3

I solved this problem by creating a custom cache backend, overriding the _base_set() function and changing the INSERT INTO statement like this. This SQL trick prevents the INSERT from happening in the case the cache_key already exists.

cursor.execute("INSERT INTO %s (cache_key, value, expires) SELECT %%s, %%s, %%s WHERE NOT EXISTS (SELECT 1 FROM %s WHERE cache_key = %%s)" % (table, table),
               [key, encoded, connections[db].ops.value_to_db_datetime(exp), key])