os.walk multiple directories at once [duplicate]

Question

Possible Duplicate:
How to join two generators in Python?

Is there a way in python to use os.walk to traverse multiple directories at once?

my_paths = []
path1 = '/path/to/directory/one/'
path2 = '/path/to/directory/two/'
for path, dirs, files in os.walk(path1, path2):
    my_paths.append(dirs)

The above example doesn't work (as os.walk only accepts one directory), but I was hoping for a more elegant solution rather than calling os.walk twice (plus then I can sort it all at once). Thanks.

Answer 1

Use itertools.chain().

for path, dirs, files in itertools.chain(os.walk(path1), os.walk(path2)):
    my_paths.append(dirs)

Answer 2

since nobody mentioned it, in this or the other referenced post:

http://docs.python.org/library/multiprocessing.html

>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
...     return x*x
...
>>> p.map(f, [1,2,3])

in this case, you'd have a list of directories. the call to map would return a list of lists from each dir, you could then choose to flatten it, or keep your results clustered

def t(p):
    my_paths = []
    for path, dirs, files in os.walk(p):
        my_paths.append(dirs)


paths = ['p1','p2','etc']
p = Pool(len(paths))
dirs = p.map(t,paths)

Answer 3

Others have mentioned itertools.chain.

There's also the option of just nesting one level more:

my_paths = []
for p in ['/path/to/directory/one/', '/path/to/directory/two/']:
    for path, dirs, files in os.walk(p):
        my_paths.append(dirs)

Answer 4

To treat multiples iterables as one, use itertools.chain:

from itertools import chain

paths = ('/path/to/directory/one/', '/path/to/directory/two/', 'etc.', 'etc.')
for path, dirs, files in chain.from_iterable(os.walk(path) for path in paths):