unable to dowload csv file from FTP server in app

Question

Im trying to read CSV files from ftp server in AppEngine, and I am able to connect to the ftp server. But it returned error when I tried to retrieve files. Here is my code to read the CSV files from server:

import ftplib
import cStringIO
import csv

session = ftplib.FTP('myftpserver.com')
session.login('username','pwd')
session.set_pasv(False)

output = cStringIO.StringIO()

session.retrbinary('RETR myfile.csv', output.write)
csvfile = csv.reader(output.getvalue().splitlines(), delimiter=',')

for i, row in enumerate(csvfile):
    print row

And here is the traceback of the error I am getting:

  Traceback (most recent call last):
  File "/home/vikas/apps/myproj/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/vikas/apps/myproj/admin/admin_actions.py", line 3528, in get_ftp_file
    session.retrbinary('RETR myfile.csv', output.write)
  File "/usr/lib/python2.7/ftplib.py", line 414, in retrbinary
    conn = self.transfercmd(cmd, rest)
  File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib/python2.7/ftplib.py", line 354, in ntransfercmd
    sock = self.makeport()
  File "/usr/lib/python2.7/ftplib.py", line 283, in makeport
    for res in socket.getaddrinfo(None, 0, self.af, socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
  File "/home/vikas/gcloud/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 318, in getaddrinfo
    raise gaierror(EAI_NONAME, 'nodename nor servname provided, or not known')
gaierror: [Errno 8] nodename nor servname provided, or not known

I have no idea what I have done wrong, none of the commands like dir() nlst() etc is working, and the above error occurred as soon as I added them .

Answer 1

Alas, the socket simulation currently offered by App Engine isn't quite good enough to cover all use cases.

Here's a sample accessing a well-known public anonymous FTP server and fetching a small text file from it, just so everybody can reproduce and experiment...: in file getit.py, we have:

import ftplib
import cStringIO

def getit():
    session = ftplib.FTP('ftp.mozilla.org')
    session.login('anonymous','')
    # session.set_pasv(False)
    session.cwd('/pub/mozilla.org')

    output = cStringIO.StringIO()
    session.retrbinary('RETR README', output.write)

    return output.getvalue()

if __name__ == '__main__':
    print(getit())

This runs fine as stand-alone, python getit.py, whether you leave the set_pasv commented as here, or remove the comment.

To embed this in a GAE app, e.g:

import getit
class GetitPage(webapp2.RequestHandler):
  def get(self):  # pylint:disable-msg=invalid-name
     try: result = getit.getit()
     except Exception as e:
         result = 'Error {}: {}'.format(type(e), e)
     self.response.headers['Content-Type'] = 'text/plain'
     self.response.out.write(result)

this works fine with the set_pasv left commented, but if you de-comment it, you get essentially the same exception you received.

So doing FTP to a server that forces you to active mode (doesn't support passive mode) is not going to work this way. However, that is a rather broken server -- could you get it fixed so that it supports the popular, default passive mode? Then your GAE app could work happily with it...!