Is there a limit to the number of values that a py

Question

I am trying to use a python set as a filter for ids from a mysql table. The python set stores all the ids to filter (about 30 000 right now) this number will grow slowly over time and I am concerned about the maximum capacity of a python set. Is there a limit to the number of elements it can contain?

Answer 1

Your biggest constraint is the amount of memory on your computer. Try the line:

s = set(xrange(10000000))

That creates a set of length 10 million, much larger than the 30,000 you give as an example. On my computer (a Macbook Air with 4GB of memory) this runs in only a few seconds. Whatever your system, it is probably you will be similarly unconstrained.

Of course, there is absolutely an upper limit, and long before that your set will start to slow down because of the number of collisions, as well as the necessary memory swapping. It thus matters how big this set will get. If you will be working with more than 10 million items, you might want to consider working with databases instead.

Answer 2

I don't know if there is an arbitrary limit for the number of items in a set. More than likely the limit is tied to the available memory.