Django ORM - LEFT OUTER JOIN with two columns?

Question

This is the relevant code:

class Book(models.Model):
    name = models.CharField(max_length=50)

class BookNote(models.Model):
    text = models.CharField(max_length=50)
    book = models.ForeignKey(Book)
    user = models.ForeignKey(settings.AUTH_USER_MODEL)
    class Meta:
        unique_together = [('book', 'user'), ]

Now, for a specific user in the website:
I want to query all the books (all the table). And,
For each book object, If the user has a BookNote for the book - get it, otherwise booknote should be null.

This is how I would do that with SQL (works):

SELECT book.name, booknote.text 
FROM book 
LEFT OUTER JOIN booknote ON 
(book.id = booknote.book_id AND booknote.user_id = {user_id_here})

This is what I've tried, does not work:

qs = Book.objects.filter(Q(booknote__user_id=user_id_here) | Q(booknote__isnull=True))

I examine qs.query and I see why - Django uses WHERE clause to filter by user_id, so I don't get all the books.
How can I do the same query with django ORM? Without raw sql?

Answer 1

The reason your query doesn't work is you're expicitly asking for either Books with your user's notes or no notes at all: this excludes books where only other users have notes.

I think what you're looking for is best performed as an annotation. Under django 2.0+, you can use the new FilteredRelation to perform a LEFT OUTER JOIN ON (... AND ...), but I had trouble doing it and maintaining the ForeignKey in the ORM; you'll have to re-export the fields you need with additional annotations.

q =  Book.objects.all().annotate(
        usernote=FilteredRelation('booknote', condition=Q(booknote__user=USER_ID)),
        usernote_text=F('usernote__text'),
        usernote_id=F('usernote'),
        )

Resulting query:

SELECT "books_book"."id", "books_book"."name", usernote."text" AS "usernote_text", usernote."id" AS "usernote_id" FROM "books_book" LEFT OUTER JOIN "books_booknote" usernote ON ("books_book"."id" = usernote."book_id" AND (usernote."user_id" = <USER_ID>))

If you're using 1.11 still, you can get the same result (but less performance and different queries) with Prefetch objects or a case-when annotation.

In models.py:

class Book(models.Model):
     # SNIP
     @property
     def usernote(self):
         # raises an error if not prefetched
         try:
             return self._usernote[0] if self._usernote else None
         except AttributeError:
             raise Exception("Book.usernote must be prefetched with prefetch_related(Book.usernote_prefetch(user)) before use")

     @staticmethod
     def usernote_prefetch(user):
         return Prefetch(
             'booknote_set',
             queryset=BookNote.objects.filter(user=user)
             to_attr='_usernote'
         )

By your query:

    q = Book.objects.all().prefetch_related(Book.usernote_prefetch(USER))

Full tests.py:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.test import TestCase
from django.db.models import *
from books.models import Book, BookNote
from django.contrib.auth import get_user_model

class BookTest(TestCase):
    def setUp(self):
        User = get_user_model()
        self.u1 = User.objects.create(username="U1")
        self.u2 = User.objects.create(username="U2")
        self.b1 = Book.objects.create(name="B1")  # Has no notes
        self.b2 = Book.objects.create(name="B2")  # Has a note for U1 and U2
        self.b3 = Book.objects.create(name="B3")  # Has a note for just U2
        self.n1 = BookNote.objects.create(text="N1", book=self.b2, user=self.u1)
        BookNote.objects.create(text="N2", book=self.b2, user=self.u2)
        BookNote.objects.create(text="N3", book=self.b1, user=self.u2)

    def test_on_multiple(self):
        q =  Book.objects.all().annotate(
                usernote=FilteredRelation('booknote', condition=Q(booknote__user=self.u1)),
                usernote_text=F('usernote__text'),
                usernote_id=F('usernote'),
                ).order_by('id')
        print(q.query)
        self.assertEqual(q.count(), Book.objects.count())
        self.assertIsNone(q[0].usernote_text)
        self.assertEqual( q[1].usernote_text, self.n1.text)
        self.assertIsNone(q[2].usernote_text)

    def test_on_multiple_prefetch(self):
        @property
        def usernote(self):
            return self._usernote[0] if self._usernote else None
        Book.usernote = usernote
        q = Book.objects.all().prefetch_related(Prefetch(
            'booknote_set',
            queryset=BookNote.objects.filter(user=self.u1),
            to_attr='_usernote'
            )).order_by('id')

        self.assertEqual(q.count(), Book.objects.count())
        self.assertIsNone(q[0].usernote)
        self.assertEqual( q[1].usernote.text, self.n1.text)
        self.assertIsNone(q[2].usernote)