Elegant way to remove items from sequence in Pytho

Question

This question already has answers here:

How to remove items from a list while iterating? (26 answers)

Closed 4 years ago.

When I am writing code in Python, I often need to remove items from a list or other sequence type based on some criteria. I haven't found a solution that is elegant and efficient, as removing items from a list you are currently iterating through is bad. For example, you can't do this:

for name in names:
    if name[-5:] == 'Smith':
        names.remove(name)

I usually end up doing something like this:

toremove = []
for name in names:
    if name[-5:] == 'Smith':
        toremove.append(name)
for name in toremove:
    names.remove(name)
del toremove

This is innefficient, fairly ugly and possibly buggy (how does it handle multiple 'John Smith' entries?). Does anyone have a more elegant solution, or at least a more efficient one?

How about one that works with dictionaries?

Answer 1

There are times when filtering (either using filter or a list comprehension) doesn't work. This happens when some other object is holding a reference to the list you're modifying and you need to modify the list in place.

for name in names[:]:
    if name[-5:] == 'Smith':
        names.remove(name)

The only difference from the original code is the use of names[:] instead of names in the for loop. That way the code iterates over a (shallow) copy of the list and the removals work as expected. Since the list copying is shallow, it's fairly quick.

Answer 2

filter would be awesome for this. Simple example:

names = ['mike', 'dave', 'jim']
filter(lambda x: x != 'mike', names)
['dave', 'jim']

Edit: Corey's list comprehension is awesome too.

Answer 3

In the case of a set.

toRemove = set([])  
for item in mySet:  
    if item is unwelcome:  
        toRemove.add(item)  
mySets = mySet - toRemove 

Answer 4

Both solutions, filter and comprehension requires building a new list. I don't know enough of the Python internals to be sure, but I think that a more traditional (but less elegant) approach could be more efficient:

names = ['Jones', 'Vai', 'Smith', 'Perez']

item = 0
while item <> len(names):
    name = names [item]
    if name=='Smith':
        names.remove(name)
    else:
        item += 1

print names

Anyway, for short lists, I stick with either of the two solutions proposed earlier.

Answer 5

The filter and list comprehensions are ok for your example, but they have a couple of problems:

• They make a copy of your list and return the new one, and that will be inefficient when the original list is really big
• They can be really cumbersome when the criteria to pick items (in your case, if name[-5:] == 'Smith') is more complicated, or has several conditions.

Your original solution is actually more efficient for very big lists, even if we can agree it's uglier. But if you worry that you can have multiple 'John Smith', it can be fixed by deleting based on position and not on value:

names = ['Jones', 'Vai', 'Smith', 'Perez', 'Smith']

toremove = []
for pos, name in enumerate(names):
    if name[-5:] == 'Smith':
        toremove.append(pos)
for pos in sorted(toremove, reverse=True):
    del(names[pos])

print names

We can't pick a solution without considering the size of the list, but for big lists I would prefer your 2-pass solution instead of the filter or lists comprehensions

Answer 6

Here is my filter_inplace implementation that can be used to filter items from a list in-place, I came up with this on my own independently before finding this page. It is the same algorithm as what PabloG posted, just made more generic so you can use it to filter lists in place, it is also able to remove from the list based on the comparisonFunc if reversed is set True; a sort-of of reversed filter if you will.

def filter_inplace(conditionFunc, list, reversed=False):
    index = 0
    while index < len(list):
        item = list[index]

        shouldRemove = not conditionFunc(item)
        if reversed: shouldRemove = not shouldRemove

        if shouldRemove:
            list.remove(item)
        else:
            index += 1