What is the purpose of __slots__
in Python — especially with respect to when I would want to use it, and when not?
相关问题
- how to define constructor for Python's new Nam
- streaming md5sum of contents of a large remote tar
- How to get the background from multiple images by
- Evil ctypes hack in python
- Correctly parse PDF paragraphs with Python
An attribute of a class instance has 3 properties: the instance, the name of the attribute, and the value of the attribute.
In regular attribute access, the instance acts as a dictionary and the name of the attribute acts as the key in that dictionary looking up value.
instance(attribute) --> value
In __slots__ access, the name of the attribute acts as the dictionary and the instance acts as the key in the dictionary looking up value.
attribute(instance) --> value
In flyweight pattern, the name of the attribute acts as the dictionary and the value acts as the key in that dictionary looking up the instance.
attribute(value) --> instance
You would want to use
__slots__
if you are going to instantiate a lot (hundreds, thousands) of objects of the same class.__slots__
only exists as a memory optimization tool.It's highly discouraged to use
__slots__
for constraining attribute creation, and in general you want to avoid it because it breaks pickle, along with some other introspection features of python.TLDR:
The special attribute
__slots__
allows you to explicitly state which instance attributes you expect your object instances to have, with the expected results:The space savings is from
__dict__
.__dict__
and__weakref__
creation if parent classes deny them and you declare__slots__
.Quick Caveats
Small caveat, you should only declare a particular slot one time in an inheritance tree. For example:
Python doesn't object when you get this wrong (it probably should), problems might not otherwise manifest, but your objects will take up more space than they otherwise should.
The biggest caveat is for multiple inheritance - multiple "parent classes with nonempty slots" cannot be combined.
To accommodate this restriction, follow best practices: Factor out all but one or all parents' abstraction which their concrete class respectively and your new concrete class collectively will inherit from - giving the abstraction(s) empty slots (just like abstract base classes in the standard library).
See section on multiple inheritance below for an example.
Requirements:
To have attributes named in
__slots__
to actually be stored in slots instead of a__dict__
, a class must inherit fromobject
.To prevent the creation of a
__dict__
, you must inherit fromobject
and all classes in the inheritance must declare__slots__
and none of them can have a'__dict__'
entry.There are a lot of details if you wish to keep reading.
Why use
__slots__
: Faster attribute access.The creator of Python, Guido van Rossum, states that he actually created
__slots__
for faster attribute access.It is trivial to demonstrate measurably significant faster access:
and
The slotted access is almost 30% faster in Python 3.5 on Ubuntu.
In Python 2 on Windows I have measured it about 15% faster.
Why use
__slots__
: Memory SavingsAnother purpose of
__slots__
is to reduce the space in memory that each object instance takes up.My own contribution to the documentation clearly states the reasons behind this:
SQLAlchemy attributes a lot of memory savings to
__slots__
.To verify this, using the Anaconda distribution of Python 2.7 on Ubuntu Linux, with
guppy.hpy
(aka heapy) andsys.getsizeof
, the size of a class instance without__slots__
declared, and nothing else, is 64 bytes. That does not include the__dict__
. Thank you Python for lazy evaluation again, the__dict__
is apparently not called into existence until it is referenced, but classes without data are usually useless. When called into existence, the__dict__
attribute is a minimum of 280 bytes additionally.In contrast, a class instance with
__slots__
declared to be()
(no data) is only 16 bytes, and 56 total bytes with one item in slots, 64 with two.For 64 bit Python, I illustrate the memory consumption in bytes in Python 2.7 and 3.6, for
__slots__
and__dict__
(no slots defined) for each point where the dict grows in 3.6 (except for 0, 1, and 2 attributes):So, in spite of smaller dicts in Python 3, we see how nicely
__slots__
scale for instances to save us memory, and that is a major reason you would want to use__slots__
.Just for completeness of my notes, note that there is a one-time cost per slot in the class's namespace of 64 bytes in Python 2, and 72 bytes in Python 3, because slots use data descriptors like properties, called "members".
Demonstration of
__slots__
:To deny the creation of a
__dict__
, you must subclassobject
:now:
Or subclass another class that defines
__slots__
and now:
but:
To allow
__dict__
creation while subclassing slotted objects, just add'__dict__'
to the__slots__
(note that slots are ordered, and you shouldn't repeat slots that are already in parent classes):and
Or you don't even need to declare
__slots__
in your subclass, and you will still use slots from the parents, but not restrict the creation of a__dict__
:And:
However,
__slots__
may cause problems for multiple inheritance:Because creating a child class from parents with both non-empty slots fails:
If you run into this problem, You could just remove
__slots__
from the parents, or if you have control of the parents, give them empty slots, or refactor to abstractions:Add
'__dict__'
to__slots__
to get dynamic assignment:and now:
So with
'__dict__'
in slots we lose some of the size benefits with the upside of having dynamic assignment and still having slots for the names we do expect.When you inherit from an object that isn't slotted, you get the same sort of semantics when you use
__slots__
- names that are in__slots__
point to slotted values, while any other values are put in the instance's__dict__
.Avoiding
__slots__
because you want to be able to add attributes on the fly is actually not a good reason - just add"__dict__"
to your__slots__
if this is required.You can similarly add
__weakref__
to__slots__
explicitly if you need that feature.Set to empty tuple when subclassing a namedtuple:
The namedtuple builtin make immutable instances that are very lightweight (essentially, the size of tuples) but to get the benefits, you need to do it yourself if you subclass them:
usage:
And trying to assign an unexpected attribute raises an
AttributeError
because we have prevented the creation of__dict__
:You can allow
__dict__
creation by leaving off__slots__ = ()
, but you can't use non-empty__slots__
with subtypes of tuple.Biggest Caveat: Multiple inheritance
Even when non-empty slots are the same for multiple parents, they cannot be used together:
Using an empty
__slots__
in the parent seems to provide the most flexibility, allowing the child to choose to prevent or allow (by adding'__dict__'
to get dynamic assignment, see section above) the creation of a__dict__
:You don't have to have slots - so if you add them, and remove them later, it shouldn't cause any problems.
Going out on a limb here: If you're composing mixins or using abstract base classes, which aren't intended to be instantiated, an empty
__slots__
in those parents seems to be the best way to go in terms of flexibility for subclassers.To demonstrate, first, let's create a class with code we'd like to use under multiple inheritance
We could use the above directly by inheriting and declaring the expected slots:
But we don't care about that, that's trivial single inheritance, we need another class we might also inherit from, maybe with a noisy attribute:
Now if both bases had nonempty slots, we couldn't do the below. (In fact, if we wanted, we could have given
AbstractBase
nonempty slots a and b, and left them out of the below declaration - leaving them in would be wrong):And now we have functionality from both via multiple inheritance, and can still deny
__dict__
and__weakref__
instantiation:Other cases to avoid slots:
__class__
assignment with another class that doesn't have them (and you can't add them) unless the slot layouts are identical. (I am very interested in learning who is doing this and why.)You may be able to tease out further caveats from the rest of the
__slots__
documentation (the 3.7 dev docs are the most current), which I have made significant recent contributions to.Critiques of other answers
The current top answers cite outdated information and are quite hand-wavy and miss the mark in some important ways.
Do not "only use
__slots__
when instantiating lots of objects"I quote:
Abstract Base Classes, for example, from the
collections
module, are not instantiated, yet__slots__
are declared for them.Why?
If a user wishes to deny
__dict__
or__weakref__
creation, those things must not be available in the parent classes.__slots__
contributes to reusability when creating interfaces or mixins.It is true that many Python users aren't writing for reusability, but when you are, having the option to deny unnecessary space usage is valuable.
__slots__
doesn't break picklingWhen pickling a slotted object, you may find it complains with a misleading
TypeError
:This is actually incorrect. This message comes from the oldest protocol, which is the default. You can select the latest protocol with the
-1
argument. In Python 2.7 this would be2
(which was introduced in 2.3), and in 3.6 it is4
.in Python 2.7:
in Python 3.6
So I would keep this in mind, as it is a solved problem.
Critique of the (until Oct 2, 2016) accepted answer
The first paragraph is half short explanation, half predictive. Here's the only part that actually answers the question
The second half is wishful thinking, and off the mark:
Python actually does something similar to this, only creating the
__dict__
when it is accessed, but creating lots of objects with no data is fairly ridiculous.The second paragraph oversimplifies and misses actual reasons to avoid
__slots__
. The below is not a real reason to avoid slots (for actual reasons, see the rest of my answer above.):It then goes on to discuss other ways of accomplishing that perverse goal with Python, not discussing anything to do with
__slots__
.The third paragraph is more wishful thinking. Together it is mostly off-the-mark content that the answerer didn't even author and contributes to ammunition for critics of the site.
Memory usage evidence
Create some normal objects and slotted objects:
Instantiate a million of them:
Inspect with
guppy.hpy().heap()
:Access the regular objects and their
__dict__
and inspect again:This is consistent with the history of Python, from Unifying types and classes in Python 2.2
You have — essentially — no use for
__slots__
.For the time when you think you might need
__slots__
, you actually want to use Lightweight or Flyweight design patterns. These are cases when you no longer want to use purely Python objects. Instead, you want a Python object-like wrapper around an array, struct, or numpy array.The class-like wrapper has no attributes — it just provides methods that act on the underlying data. The methods can be reduced to class methods. Indeed, it could be reduced to just functions operating on the underlying array of data.
A very simple example of
__slot__
attribute.Problem: Without
__slots__
If I don't have
__slot__
attribute in my class, I can add new attributes to my objects.If you look at example above, you can see that obj1 and obj2 have their own x and y attributes and python has also created a
dict
attribute for each object (obj1 and obj2).Suppose if my class Test has thousands of such objects? Creating an additional attribute
dict
for each object will cause lot of overhead (memory, computing power etc.) in my code.Solution: With
__slots__
Now in the following example my class Test contains
__slots__
attribute. Now I can't add new attributes to my objects (except attributex
) and python doesn't create adict
attribute anymore. This eliminates overhead for each object, which can become significant if you have many objects.