How can I, in python, iterate over multiple 2d lis

Question

If I'm making a simple grid based game, for example, I might have a few 2d lists. One might be for terrain, another might be for objects, etc. Unfortunately, when I need to iterate over the lists and have the contents of a square in one list affect part of another list, I have to do something like this.

for i in range(len(alist)):
    for j in range(len(alist[i])):
        if alist[i][j].isWhatever:
            blist[i][j].doSomething()

Is there a nicer way to do something like this?

Answer 1

Generator expressions and izip from itertools module will do very nicely here:

from itertools import izip
for a, b in (pair for (aline, bline) in izip(alist, blist) 
             for pair in izip(aline, bline)):
    if a.isWhatever:
        b.doSomething()

The line in for statement above means:

• take each line from combined grids alist and blist and make a tuple from them (aline, bline)
• now combine these lists with izip again and take each element from them (pair).

This method has two advantages:

• there are no indices used anywhere
• you don't have to create lists with zip and use more efficient generators with izip instead.

Answer 2

If anyone is interested in performance of the above solutions, here they are for 4000x4000 grids, from fastest to slowest:

• Brian: 1.08s (modified, with izip instead of zip)
• John: 2.33s
• DzinX: 2.36s
• ΤΖΩΤΖΙΟΥ: 2.41s (but object initialization took 62s)
• Eugene: 3.17s
• Robert: 4.56s
• Brian: 27.24s (original, with zip)

EDIT: Added Brian's scores with izip modification and it won by a large amount!

John's solution is also very fast, although it uses indices (I was really surprised to see this!), whereas Robert's and Brian's (with zip) are slower than the question creator's initial solution.

So let's present Brian's winning function, as it is not shown in proper form anywhere in this thread:

from itertools import izip
for a_row,b_row in izip(alist, blist):
    for a_item, b_item in izip(a_row,b_row):
        if a_item.isWhatever:
            b_item.doSomething()

Answer 3

When you are operating with grids of numbers and want really good performance, you should consider using Numpy. It's surprisingly easy to use and lets you think in terms of operations with grids instead of loops over grids. The performance comes from the fact that the operations are then run over whole grids with optimised SSE code.

For example here is some numpy using code that I wrote that does brute force numerical simulation of charged particles connected by springs. This code calculates a timestep for a 3d system with 100 nodes and 99 edges in 31ms. That is over 10x faster than the best pure python code I could come up with.

from numpy import array, sqrt, float32, newaxis
def evolve(points, velocities, edges, timestep=0.01, charge=0.1, mass=1., edgelen=0.5, dampen=0.95):
    """Evolve a n body system of electrostatically repulsive nodes connected by
       springs by one timestep."""
    velocities *= dampen

    # calculate matrix of distance vectors between all points and their lengths squared
    dists = array([[p2 - p1 for p2 in points] for p1 in points])
    l_2 = (dists*dists).sum(axis=2)

    # make the diagonal 1's to avoid division by zero
    for i in xrange(points.shape[0]):
        l_2[i,i] = 1

    l_2_inv = 1/l_2
    l_3_inv = l_2_inv*sqrt(l_2_inv)

    # repulsive force: distance vectors divided by length cubed, summed and multiplied by scale
    scale = timestep*charge*charge/mass
    velocities -= scale*(l_3_inv[:,:,newaxis].repeat(points.shape[1], axis=2)*dists).sum(axis=1)

    # calculate spring contributions for each point
    for idx, (point, outedges) in enumerate(izip(points, edges)):
        edgevecs = point - points.take(outedges, axis=0)
        edgevec_lens = sqrt((edgevecs*edgevecs).sum(axis=1))
        scale = timestep/mass
        velocities[idx] += (edgevecs*((((edgelen*scale)/edgevec_lens - scale))[:,newaxis].repeat(points.shape[1],axis=1))).sum(axis=0)

    # move points to new positions
    points += velocities*timestep

Answer 4

If a.isWhatever is rarely true you could build an "index" once:

a_index = set((i,j) 
              for i,arow in enumerate(a) 
              for j,a in enumerate(arow) 
              if a.IsWhatever())

and each time you want something to be done:

for (i,j) in a_index:
    b[i][j].doSomething()

If a changes over time, then you will need to keep the index up-to-date. That's why I used a set, so items can be added and removed fast.

Answer 5

As a slight style change, you could use enumerate:

for i, arow in enumerate(alist):
    for j, aval in enumerate(arow):
        if aval.isWhatever():
            blist[i][j].doSomething()

I don't think you'll get anything significantly simpler unless you rearrange your data structures as Federico suggests. So that you could turn the last line into something like "aval.b.doSomething()".

Answer 6

If the two 2D-lists remain constant during the lifetime of your game and you can't enjoy Python's multiple inheritance to join the alist[i][j] and blist[i][j] object classes (as others have suggested), you could add a pointer to the corresponding b item in each a item after the lists are created, like this:

for a_row, b_row  in itertools.izip(alist, blist):
    for a_item, b_item in itertools.izip(a_row, b_row):
        a_item.b_item= b_item

Various optimisations can apply here, like your classes having __slots__ defined, or the initialization code above could be merged with your own initialization code e.t.c. After that, your loop will become:

for a_row in alist:
    for a_item in a_row:
        if a_item.isWhatever():
            a_item.b_item.doSomething()

That should be more efficient.