mapping a (lambda) function to lists of strings fa

Question

I try to understand where my Python script goes awry. I have a pandas Series (diagnoses) of lists, each a list of strings (never empty). I can and did verify this with diagnoses.map(type) and

for x in diagnoses[0]:
    type x

Yet when I would map a lambda function to this Series of lists, I get a TypeError: 'float' object not iterable.

Imagine the data looking like this:

LopNr   AR  var3    va4 var5    var6    var7    var8    var9    var10   DIAGNOS
6   2011                                    S834
6   2011                                    K21 S834

And the code is:

from pandas import *
tobacco = lambda lst: any( (((x >= 'C30') and (x<'C40')) or ((x >= 'F17') and (x<'F18')))  for x in lst)
treatments = read_table(filename,usecols=[0,1,10])
diagnoses = treatments['DIAGNOS'].str.split(' ')
treatments['tobacco'] = diagnoses.map(tobacco)

What is going on, and how can I fix this?

PS: The same code definitely runs on a very similar Series if I import the source text file with IOpro first and build a dataframe from that adapter, see below. I am not sure why that would change the relevant datatypes, as far as I could verify the pandas Series has lists of strings in either case… This is with Python 2.7.6 and pandas 0.13.1.

import iopro
adapter = iopro.text_adapter(filename,parser='csv',field_names=True,output='dataframe',delimiter='\t')
treatments = adapter[['LopNr','AR','DIAGNOS']][:]

Answer 1

The TypeError: 'float' object is not iterable could happen if the data is missing a value for DIAGNOS. For example, when data looks like this:

LopNr   AR  var3    va4 var5    var6    var7    var8    var9    var10   DIAGNOS
6   2011    a   a   a   a   a   a   a   a   S834
6   2011    a   a   a   a   a   a   a   a   
6   2011    a   a   a   a   a   a   a   a   K21 S834

Then

    In [68]: treatments = pd.read_table('data', usecols=[0,1,10])

In [69]: treatments
Out[69]: 
       LopNr    AR   DIAGNOS
0          6  2011      S834
1          6  2011       NaN
2          6  2011  K21 S834

[3 rows x 3 columns]

The NaN in the DIAGNOS column is the source of the problem, since str.split(' ') preserves the NaN:

In [70]: diagnoses = treatments['DIAGNOS'].str.split(' ')

In [71]: diagnoses
Out[72]: 
0         [S834]
1            NaN
2    [K21, S834]
Name: DIAGNOS, dtype: object

The NaN gets passed to the tobacco function when diganose.map(tobacco) is called. Since NaN is a float and not iterable, the for x in lst loop raises the TypeError.

---

To avoid this error, replace the NaNs in treatments['DIAGNOS']:

import pandas as pd

def tobacco(lst):
    return any((('C30' <= x < 'C40') or ('F17' <= x <'F18')) for x in lst)

treatments = pd.read_table('data', usecols=[0,1,10])
treatments['DIAGNOS'].fillna('', inplace=True)
diagnoses = treatments['DIAGNOS'].str.split(' ')
treatments['tobacco'] = diagnoses.map(tobacco)
print(treatments)

yields

       LopNr    AR   DIAGNOS tobacco
0          6  2011      S834   False
1          6  2011             False
2          6  2011  K21 S834   False

[3 rows x 4 columns]