loading a dataset in python (numpy) when there are

Question

I have a big dataset contains numeric data and in some of its rows there are variable spaces delimiting columns, like:

4 5 6
7  8    9
2 3 4

When I use this line:

dataset=numpy.loadtxt("dataset.txt", delimiter=" ")

I get this error:

ValueError: Wrong number of columns at line 2

How can I change the code to ignore multiple spaces as well?

Answer 1

The default for delimiter is 'any whitespace'. If you leave loadtxt out, it copes with multiple spaces.

>>> from io import StringIO
>>> dataset = StringIO('''\
... 4 5 6
... 7 8     9
... 2 3 4''')
>>> import numpy
>>> dataset_as_numpy = numpy.loadtxt(dataset)
>>> dataset_as_numpy
array([[ 4.,  5.,  6.],
       [ 7.,  8.,  9.],
       [ 2.,  3.,  4.]])

Answer 2

Use the numpy.genfromtxt function:

>>> import numpy as np
>>> dataset = np.genfromtxt(dataset.txt)
>>> print dataset
array([[ 4., 5., 6.], [ 7., 8., 19.], [ 2., 3., 4.], [ 1., 3., 204.]])

This is from the numpy documentation:

By default, genfromtxt assumes delimiter=None, meaning that the line is split along white spaces (including tabs) and that consecutive white spaces are considered as a single white space.

Hope this helps!