Restrict the initial queryset to objects in pagina

Question

I am defining a ModelViewSet using django-rest-framework.
I need to override the default queryset to perform some processing on the queryset objects before rendering the response.

This process is time-expensive so I would like to execute it only on the objects that will be actually available to the consumer due to the paginated response, instead of applying this process to ALL the objects and applying pagination AFTER finishing my processing, which I can notice (correct me if I'm wrong) is the default behavior in DRF.

---

In short what I need is:

If the default queryset is 1000 objects, but the pagination is restricted to 25 objects per page, I want to apply my process only those 25 objects. Please note there is no other constraints for reducing the final amount of objects other than pagination.

Is there a way to do this?
Is overriding the default queryset a bad idea in this case?

Thanks!

Answer 1

Theory:

As it has been stated before:

• Django REST Framework Return Only Objects Needed For Pagination & Total Count
• Django lazy QuerySet and pagination

Django querysets are lazy.
They only hit the database when they absolutely need to (like when you are doing to do the processing before the query and pagination.).

There are two parts on DRF's pagination process:

• The paginate_queryset
• The get_paginated_response

We can choose which part to override, depending on our needs (See the Practice part)

---

Practice:

Depending on what you really want to process, there are two options.

For this I assume that we are going to extend/override the LimitOffsetPagination class, which is easier for an example, but the same principles apply to every other DRF pagination.

1. Processing the model objects:

   If you want the preprocessing to be executed on the model objects and be permanent on your database, you need to override the paginate_queryset method:

   class MyPaginationMethod(LimitOffsetPagination):
   
       def paginate_queryset(self, queryset, request, view=None):
           self.count = _get_count(queryset)
           self.limit = self.get_limit(request)
           if self.limit is None:
               return None
   
           self.offset = self.get_offset(request)
           self.request = request
           if self.count > self.limit and self.template is not None:
           self.display_page_controls = True
   
           if self.count == 0 or self.offset > self.count:
               return []
   
           """
           Do your processing here on the  
           queryset[self.offset:self.offset + self.limit]
           Which has actually self.limit (e.g 25) amount of objects.
           """
   
           return list(YOUR_PROCESSED_QUERYSET)
   

2. Processing the paginated response:

   If you want the preprocessing to be executed on the response and NOT to be permanent on your database, you need to override the get_paginated_response method:

   class MyPaginationMethod(LimitOffsetPagination):
   
       def get_paginated_response(self, data):
           """
           Do your processing here on the data variable.
           The data is a list of OrderedDicts containing every object's
           representation as a dict.
           """
           return Response(OrderedDict([
               ('count', self.count),
               ('next', self.get_next_link()),
               ('previous', self.get_previous_link()),
               ('results', YOUR_PROCESSED_DATA)
           ]))
   

Answer 2

There is no "easy" way to do that. In Django REST framework pagination is done in the same method as rendering.

So I guess the best way to go is to define your own Viewset and redeclare the list method:

from rest_framework.viewssets import ModelViewSet

class MyModelViewSet(ModelViewSet):

  def list(self, request, *args, **kwargs):
    self.object_list = self.filter_queryset(self.get_queryset())                
    if not self.allow_empty and not self.object_list:                           
      warnings.warn(                                                          
        'The `allow_empty` parameter is due to be deprecated. '             
        'To use `allow_empty=False` style behavior, You should override '   
        '`get_queryset()` and explicitly raise a 404 on empty querysets.',  
        PendingDeprecationWarning
      )              
      class_name = self.__class__.__name__                                    
      error_msg = self.empty_error % {'class_name': class_name}
      raise Http404(error_msg)                                                

    page = self.paginate_queryset(self.object_list)                             


    ## INSERT YOUR CODE HERE


    if page is not None:
      serializer = self.get_pagination_serializer(page)                       
    else:       
      serializer = self.get_serializer(self.object_list, many=True)           

    return Response(serializer.data)