I'm working on a webapp with a Django backend that provides a Django REST Framework API for the frontend to consume. I've recently run into some performance issues, so I've started investigating the performance of each of my endpoints- and most of them are issuing far too many queries to the database.
I've got several prefetch-related issues that I can't figure out, but this is (I think) the simplest. I've implemented select_related
and prefetch_related
where I can, thanks to the docs as well as this excellent post on how to think about eager loading.
For one particular model, I've reduced the number of queries decently, but I can't figure out why I still have certain duplicates.
models.py:
class ReadingGroup(models.model):
owner = models.ForeignKeyField(settings.AUTH_USER_MODEL)
users = models.ManyToManyField(settings.AUTH_USER_MODEL)
book_type = models.ForeignKeyField(BookType)
....
<other group related fields>
def __str__(self):
return '%s group: %s' % (self.name, self.book_type)
serializers.py:
class ReadingGroupSerializer(serializers.ModelSerializer):
users = UserSerializer(many = True,read_only=True)
owner = UserSerializer(read_only=True)
class Meta:
model = ReadingGroup
fields = ('url', 'id','owner', 'users')
@staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('owner')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('users')
return queryset
views.py:
class ReadingGroupViewset(views.ModelViewset):
def get_queryset(self):
qs = ReadingGroup.objects.all()
qs = self.get_serializer_class().setup_eager_loading(qs)
return qs
urls.py
router.register(r'groups', ReadingGroupViewset)
The setup_eager_loading()
method reduced the number of queries to retrieve all 12 ReadingGroup
instances in my database from 40 to 17- with 12 queries being duplicates. Each of the duplicate queries is a result of the model __str__
method being called separately on each model instance.
I initially thought it was due to something in the DRF docs that suggest the __str__
method is only called for use in the browsable API:
The built-in str method of the model will be used to generate string representations of the objects used to populate the choices property. These choices are used to populate select HTML inputs in the browsable API.
However, this doesn't seem to be the case. I implemented a workaround for the Django debug toolbar to wrap JSON responses in HTML (since the toolbar only profiles HTML responses), added ?format=json
to the endpoint request (so the browsable API is circumvented), but the duplicate queries are still issued.
Is DRF calling the __str__
method despite not using the browsable API for requests, or is there some other issue with my model?
Could this be due to behavior related to using a ModelViewset
? The endpoint I'm hitting for this test is just /api/groups/
, which is one of the URLs automatically generated by the register()
command.
EDIT: The comment from @serg below got me thinking- if it's querying a field that isn't in the serializer, I suppose it doesn't hurt to prefetch that as well. Adding the book_type
to setup_eager_loading
got rid of the duplicate queries:
@staticmethod
def setup_eager_loading(queryset):
#select_related for 'to-one' relationships
queryset = queryset.select_related('owner')
queryset = queryset.select_related('book_type')
#prefetch_related for 'to-many' relationships
queryset = queryset.prefetch_related('users')
return queryset
I added that field to the code above so it's more clear that the __str__
method was accessing a relational field.