How to combine 2 or more querysets in a Django vie

Question

I am trying to build the search for a Django site I am building, and in the search I am searching in 3 different models. And to get pagination on the search result list I would like to use a generic object_list view to display the results. But to do that i have to merge 3 querysets into one.

How can i do that? I've tried this:

result_list = []            
page_list = Page.objects.filter(
    Q(title__icontains=cleaned_search_term) | 
    Q(body__icontains=cleaned_search_term))
article_list = Article.objects.filter(
    Q(title__icontains=cleaned_search_term) | 
    Q(body__icontains=cleaned_search_term) | 
    Q(tags__icontains=cleaned_search_term))
post_list = Post.objects.filter(
    Q(title__icontains=cleaned_search_term) | 
    Q(body__icontains=cleaned_search_term) | 
    Q(tags__icontains=cleaned_search_term))

for x in page_list:
    result_list.append(x)
for x in article_list:
    result_list.append(x)
for x in post_list:
    result_list.append(x)

return object_list(
    request, 
    queryset=result_list, 
    template_object_name='result',
    paginate_by=10, 
    extra_context={
        'search_term': search_term},
    template_name="search/result_list.html")

But this doesn't work I get an error when I try to use that list in the generic view. The list is missing the clone attribute.

Anybody know how I can merge the three lists, page_list, article_list and post_list?

Answer 1

In case you want to chain a lot of querysets, try this:

from itertools import chain
result = list(chain(*docs))

where: docs is a list of querysets

Answer 2

The big downside of your current approach is its inefficiency with large search result sets, as you have to pull down the entire result set from the database each time, even though you only intend to display one page of results.

In order to only pull down the objects you actually need from the database, you have to use pagination on a QuerySet, not a list. If you do this, Django actually slices the QuerySet before the query is executed, so the SQL query will use OFFSET and LIMIT to only get the records you will actually display. But you can't do this unless you can cram your search into a single query somehow.

Given that all three of your models have title and body fields, why not use model inheritance? Just have all three models inherit from a common ancestor that has title and body, and perform the search as a single query on the ancestor model.

Answer 3

Related, for mixing querysets from the same model, or for similar fields from a few models, Starting with Django 1.11 a qs.union() method is also available:

union()

union(*other_qs, all=False)

New in Django 1.11. Uses SQL’s UNION operator to combine the results of two or more QuerySets. For example:

>>> qs1.union(qs2, qs3)

The UNION operator selects only distinct values by default. To allow duplicate values, use the all=True argument.

union(), intersection(), and difference() return model instances of the type of the first QuerySet even if the arguments are QuerySets of other models. Passing different models works as long as the SELECT list is the same in all QuerySets (at least the types, the names don’t matter as long as the types in the same order).

In addition, only LIMIT, OFFSET, and ORDER BY (i.e. slicing and order_by()) are allowed on the resulting QuerySet. Further, databases place restrictions on what operations are allowed in the combined queries. For example, most databases don’t allow LIMIT or OFFSET in the combined queries.

https://docs.djangoproject.com/en/1.11/ref/models/querysets/#django.db.models.query.QuerySet.union

Answer 4

For searching it's better to use dedicated solutions like Haystack - it's very flexible.

Answer 5

Requirements: Django==2.0.2, django-querysetsequence==0.8

In case you want to combine querysets and still come out with a QuerySet, you might want to check out django-queryset-sequence.

But one note about it. It only takes two querysets as it's argument. But with python reduce you can always apply it to multiple querysets.

from functools import reduce
from queryset_sequence import QuerySetSequence

combined_queryset = reduce(QuerySetSequence, list_of_queryset)

And that's it. Below is a situation I ran into and how I employed list comprehension, reduce and django-queryset-sequence

from functools import reduce
from django.shortcuts import render    
from queryset_sequence import QuerySetSequence

class People(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    mentor = models.ForeignKey('self', null=True, on_delete=models.SET_NULL, related_name='my_mentees')

class Book(models.Model):
    name = models.CharField(max_length=20)
    owner = models.ForeignKey(Student, on_delete=models.CASCADE)

# as a mentor, I want to see all the books owned by all my mentees in one view.
def mentee_books(request):
    template = "my_mentee_books.html"
    mentor = People.objects.get(user=request.user)
    my_mentees = mentor.my_mentees.all() # returns QuerySet of all my mentees
    mentee_books = reduce(QuerySetSequence, [each.book_set.all() for each in my_mentees])

    return render(request, template, {'mentee_books' : mentee_books})

Answer 6

Try this:

matches = pages | articles | posts

Retains all the functions of the querysets which is nice if you want to order_by or similar.

Oops, please note that this doesn't work on querysets from two different models...