mypy: how to define a generic subclass

Question

I have a subclass of queue.Queue like so:

class SetQueue(queue.Queue):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize=0):
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()

    def _put(self):
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)

I am trying to use mypy for static type checking. In this case, the SetQueue should take a generic object T. This is my attempt so far:

from typing import Generic, Iterable, Set, TypeVar

# Type for mypy generics
T = TypeVar('T')

class SetQueue(queue.Queue):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)

mypy throws a warning on the class definition line saying "Missing type parameters for generic type".

I think that I need a Generic[T] somewhere but every attempt that I have made throws a syntax error. All of the examples in the docs show subclassing from Generic[T] but don't subclass from any other object.

Does anyone know how to define the generic type for SetQueue?

Answer 1

The problem here is that queue.Queue does not actually not inherit from typing.Generic, but the typeshed stubs for it says that it does. This is a bit of a necessary evil until the stdlib fully buys into typing, if ever. As a result, the actual queue.Queue does not have the typing.GenericMeta metaclass that gives generic classes their __getitem__ ability at runtime:

For example, this code type-checks ok in mypy, but fails at runtime:

from typing import Generic, Iterable, Set, TypeVar, TYPE_CHECKING
import queue

# Type for mypy generics
T = TypeVar('T')


class SetQueue(queue.Queue[T]):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)


my_queue = queue.Queue()  # type: queue.Queue[int]
my_queue.put(1)
my_queue.put('foo')  # error

my_set_queue = SetQueue()  # type: SetQueue[int]
my_set_queue.put(1)
my_set_queue.put('foo')  # error

The error raised is TypeError: 'type' object is not subscriptable, meaning that queue.Queue[T] (i.e. queue.Queue.__getitem__) is not supported.

Here's a hack to make it work at runtime as well:

from typing import Generic, Iterable, Set, TypeVar, TYPE_CHECKING
import queue

# Type for mypy generics
T = TypeVar('T')

if TYPE_CHECKING:
    Queue = queue.Queue
else:
    class FakeGenericMeta(type):
        def __getitem__(self, item):
            return self

    class Queue(queue.Queue, metaclass=FakeGenericMeta):
        pass


class SetQueue(Queue[T]):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)


my_queue = queue.Queue()  # type: queue.Queue[int]
my_queue.put(1)
my_queue.put('foo')  # error

my_set_queue = SetQueue()  # type: SetQueue[int]
my_set_queue.put(1)
my_set_queue.put('foo')  # error

There may be a better way to patch in the metaclass. I'm curious to know if anyone comes up with a more elegant solution.

Edit: I should note that multiple inheritance did not work because class SetQueue(queue.Queue, Generic[T]) fails to relate SetQueue's T to queue.Queue's