Reading tokens from a file in python 3.x

Question

Is there some way to read in information from a file by token regardless of the formatting? For example, if I am trying to generate a ppm image from an input file and instead of 2 pixels being

255 0 0
0 0 255

it appears as

255
0 0
0
0 255

I'd like to read by token in this situation instead of by line in my loop but I can't find any built in methods regarding this.

Answer 1

You can use chain.from_iterable, where the iter-able would be line.split() for line in fin:

>>> with open('temp.txt', 'r') as fin:
...      iter = chain.from_iterable(line.split() for line in fin)
...      print(list(iter))
... 
['255', '0', '0', '0', '0', '255']

Answer 2

You could always roll your own file iterator:

class file_tokens:
    def __init__(self, file):
        self.file = file
        self.line = []
    def __iter__(self):
        return self
    def next(self):
        while not len(self.line):
            self.line = self.file.readline()
            if not self.line:
                raise StopIteration
            self.line = self.line.split()
        return self.line.pop(0)

Then use like a normal file:

for token in file_tokens(open(infile)):
    print('Token: ' + token)