Download CSV directly into Python CSV parser

Question

I'm trying to download CSV content from morningstar and then parse its contents. If I inject the HTTP content directly into Python's CSV parser, the result is not formatted correctly. Yet, if I save the HTTP content to a file (/tmp/tmp.csv), and then import the file in the python's csv parser the result is correct. In other words, why does:

def finDownload(code,report):
    h = httplib2.Http('.cache')
    url = 'http://financials.morningstar.com/ajax/ReportProcess4CSV.html?t=' + code + '&region=AUS&culture=en_us&reportType='+ report + '&period=12&dataType=A&order=asc&columnYear=5&rounding=1&view=raw&productCode=usa&denominatorView=raw&number=1'
    headers, data = h.request(url)
    return data

balancesheet = csv.reader(finDownload('FGE','is'))
for row in balancesheet:
    print row

return:

['F']
['o']
['r']
['g']
['e']
[' ']
['G']
['r']
['o']
['u']
     (etc...)

instead of:

[Forge Group Limited (FGE) Income Statement']

?

Answer 1

The problem results from the fact that iteration over a file is done line-by-line whereas iteration over a string is done character-by-character.

You want StringIO/cStringIO (Python 2) or io.StringIO (Python 3, thanks to John Machin for pointing me to it) so a string can be treated as a file-like object:

Python 2:

mystring = 'a,"b\nb",c\n1,2,3'
import cStringIO
csvio = cStringIO.StringIO(mystring)
mycsv = csv.reader(csvio)

Python 3:

mystring = 'a,"b\nb",c\n1,2,3'
import io
csvio = io.StringIO(mystring, newline="")
mycsv = csv.reader(csvio)

Both will correctly preserve newlines inside quoted fields:

>>> for row in mycsv: print(row)
...
['a', 'b\nb', 'c']
['1', '2', '3']