Python: Adding element to list while iterating

Question

I know that it is not allowed to remove elements while iterating a list, but is it allowed to add elements to a python list while iterating. Here is an example:

    for a in myarr:
      if somecond(a):
          myarr.append(newObj())

I have tried this in my code and it seems to work fine, however I don't know if it's because I am just lucky and that it will break at some point in the future?

EDIT: I prefer not to copy the list since "myarr" is huge, and therefore it would be too slow. Also I need to check the appended objects with "somecond()".

EDIT: At some point "somecond(a)" will be false, so there can not be an infinite loop.

EDIT: Someone asked about the "somecond()" function. Each object in myarr has a size, and each time "somecond(a)" is true and a new object is appended to the list, the new object will have a size smaller than a. "somecond()" has an epsilon for how small objects can be and if they are too small it will return "false"

Answer 1

make copy of your original list, iterate over it, see the modified code below

for a in myarr[:]:
      if somecond(a):
          myarr.append(newObj())

Answer 2

well, according to http://docs.python.org/tutorial/controlflow.html

It is not safe to modify the sequence being iterated over in the loop (this can only happen for mutable sequence types, such as lists). If you need to modify the list you are iterating over (for example, to duplicate selected items) you must iterate over a copy.

Answer 3

You can do this.

bonus_rows = []
for a in myarr:
  if somecond(a):
      bonus_rows.append(newObj())
myarr.extend( bonus_rows )

Answer 4

In short: If you'are absolutely sure all new objects fail somecond() check, then your code works fine, it just wastes some time iterating the newly added objects.

Before giving a proper answer, you have to understand why it considers a bad idea to change list/dict while iterating. When using for statement, Python tries to be clever, and returns a dynamically calculated item each time. Take list as example, python remembers a index, and each time it returns l[index] to you. If you are changing l, the result l[index] can be messy.

NOTE: Here is a stackoverflow question to demonstrate this.

The worst case for adding element while iterating is infinite loop, try(or not if you can read a bug) the following in a python REPL:

import random

l = [0]
for item in l:
    l.append(random.randint(1, 1000))
    print item

It will print numbers non-stop until memory is used up, or killed by system/user.

Understand the internal reason, let's discuss the solutions. Here are a few:

1. make a copy of origin list

Iterating the origin list, and modify the copied one.

result = l[:]
for item in l:
    if somecond(item):
        result.append(Obj())

2. control when the loop ends

Instead of handling control to python, you decides how to iterate the list:

length = len(l)
for index in range(length):
    if somecond(l[index]):
        l.append(Obj())

Before iterating, calculate the list length, and only loop length times.

3. store added objects in a new list

Instead of modifying the origin list, store new object in a new list and concatenate them afterward.

added = [Obj() for item in l if somecond(item)]
l.extend(added)

Answer 5

I had a similar problem today. I had a list of items that needed checking; if the objects passed the check, they were added to a result list. If they didn't pass, I changed them a bit and if they might still work (size > 0 after the change), I'd add them on to the back of the list for rechecking.

I went for a solution like

items = [...what I want to check...]
result = []
while items:
    recheck_items = []
    for item in items:
        if check(item):
            result.append(item)
        else:
            item = change(item)  # Note that this always lowers the integer size(),
                                 # so no danger of an infinite loop
            if item.size() > 0:
                recheck_items.append(item)
    items = recheck_items  # Let the loop restart with these, if any

My list is effectively a queue, should probably have used some sort of queue. But my lists are small (like 10 items) and this works too.

Answer 6

You could use the islice from itertools to create an iterator over a smaller portion of the list. Then you can append entries to the list without impacting the items you're iterating over:

islice( myarr, 0, len(myarr)-1 )

Even better, you don't even have to iterate over all the elements. You can increment a step size.