Caching queryset choices for ModelChoiceField or M

Question

When using ModelChoiceField or ModelMultipleChoiceField in a Django form, is there a way to pass in a cached set of choices? Currently, if I specify the choices via the queryset parameter, it results in a database hit.

I'd like to cache these choices using memcached and prevent unnecessary hits to the database when displaying a form with such a field.

Answer 1

You can override "all" method in QuerySet something like

from django.db import models
class AllMethodCachingQueryset(models.query.QuerySet):
    def all(self, get_from_cache=True):
        if get_from_cache:
            return self
        else:
            return self._clone()


class AllMethodCachingManager(models.Manager):
    def get_query_set(self):
        return AllMethodCachingQueryset(self.model, using=self._db)


class YourModel(models.Model):
    foo = models.ForeignKey(AnotherModel)

    cache_all_method = AllMethodCachingManager()

And then change queryset of field before form using (for exmple when you use formsets)

form_class.base_fields['foo'].queryset = YourModel.cache_all_method.all()

Answer 2

Here is a little hack I use with Django 1.10 to cache a queryset in a formset:

qs = my_queryset

# cache the queryset results
cache = [p for p in qs]

# build an iterable class to override the queryset's all() method
class CacheQuerysetAll(object):
    def __iter__(self):
        return iter(cache)
    def _prefetch_related_lookups(self):
        return False
qs.all = CacheQuerysetAll

# update the forms field in the formset 
for form in formset.forms:
    form.fields['my_field'].queryset = qs

Answer 3

@jnns I noticed that in your code the queryset is evaluated twice (at least in my Admin inline context), which seems to be an overhead of django admin anyway, even without this mixin (plus one time per inline when you don't have this mixing).

In the case of this mixin, this is due to the fact that formfield.choices has a setter that (to simplify) triggers the re-evaluation of the object's queryset.all()

I propose an improvement which consists of dealing directly with formfield.cache_choices and formfield.choice_cache

Here it is:

class ForeignKeyCacheMixin(object):

    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super(ForeignKeyCacheMixin, self).formfield_for_foreignkey(db_field, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        formfield.cache_choices = True
        if db_field.name in cache:
            formfield.choice_cache = cache[db_field.name]
        else:
            formfield.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield

Answer 4

The reason that ModelChoiceField in particular creates a hit when generating choices - regardless of whether the QuerySet has been populated previously - lies in this line

for obj in self.queryset.all(): 

in django.forms.models.ModelChoiceIterator. As the Django documentation on caching of QuerySets highlights,

callable attributes cause DB lookups every time.

So I'd prefer to just use

for obj in self.queryset:

even though I'm not 100% sure about all implications of this (I do know I do not have big plans with the queryset afterwards, so I think I'm fine without the copy .all() creates). I'm tempted to change this in the source code, but since I'm going to forget about it at the next install (and it's bad style to begin with) I ended up writing my custom ModelChoiceField:

class MyModelChoiceIterator(forms.models.ModelChoiceIterator):
    """note that only line with # *** in it is actually changed"""
    def __init__(self, field):
        forms.models.ModelChoiceIterator.__init__(self, field)

    def __iter__(self):
        if self.field.empty_label is not None:
            yield (u"", self.field.empty_label)
        if self.field.cache_choices:
            if self.field.choice_cache is None:
                self.field.choice_cache = [
                    self.choice(obj) for obj in self.queryset.all()
                ]
            for choice in self.field.choice_cache:
                yield choice
        else:
            for obj in self.queryset: # ***
                yield self.choice(obj)


class MyModelChoiceField(forms.ModelChoiceField):
    """only purpose of this class is to call another ModelChoiceIterator"""
    def __init__(*args, **kwargs):
        forms.ModelChoiceField.__init__(*args, **kwargs)

    def _get_choices(self):
        if hasattr(self, '_choices'):
            return self._choices

        return MyModelChoiceIterator(self)

    choices = property(_get_choices, forms.ModelChoiceField._set_choices)

This does not solve the general problem of database caching, but since you're asking about ModelChoiceField in particular and that's exactly what got me thinking about that caching in the first place, thought this might help.

Answer 5

I also stumbled over this problem while using an InlineFormset in the Django Admin that itself referenced two other Models. A lot of unnecessary queries are generated because, as Nicolas87 explained, ModelChoiceIterator fetches the queryset everytime from scratch.

The following Mixin can be added to admin.ModelAdmin, admin.TabularInline or admin.StackedInline to reduce the number of queries to just the ones needed to fill the cache. The cache is tied to the Request object, so it invalidates on a new request.

 class ForeignKeyCacheMixin(object):
    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super(ForeignKeyCacheMixin, self).formfield_for_foreignkey(db_field, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        if cache.get(db_field.name):
            formfield.choices = cache[db_field.name]
        else:
            formfield.choices.field.cache_choices = True
            formfield.choices.field.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield