Python FTP implicit TLS connection issue

Question

I have a need to connect to FTPS server to which I am able to connect successfully using lftp. However, when I try with Python ftplib.FTP_TLS, it times out, the stack trace shows that it is waiting for the server to send welcome message or like. Does anyone know what the issue is and how to overcome? I wonder if there is something needs to be done on server side, but how come lftp client is working fine. Any help is greatly appreciated.

Here is the stack trace:

    ftp = ftplib.FTP_TLS()  
    ftp.connect(cfg.HOST, cfg.PORT, timeout=60)
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 135, in connect  
    self.welcome = self.getresp()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 210, in getresp  
    resp = self.getmultiline()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 196, in getmultiline  
    line = self.getline()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 183, in getline  
    line = self.file.readline()  
  File "C:\Users\username\Softwares\Python27\lib\socket.py", line 447, in readline  
    data = self._sock.recv(self._rbufsize)  
socket.timeout: timed out  

A successful login using lftp to the same ftps server:

$ lftp
lftp :~> open ftps://ip_address:990
lftp ip_address:~> set ftps:initial-prot P
lftp ip_address:~> login ftps_user_id  ftps_user_passwd
lftp sftp_user_id@ip_address:~> ls
ls: Fatal error: SSL_connect: self signed certificate
lftp ftps_user_id@ip_address:~> set ssl:verif-certificate off
lftp ftps_user_id@ip_address:~> ls
lftp ftps_user_id@ip_address:/>

BTW, I am using Python 2.7.3. I did quite a bit of search using Google but have not found anything helpful.

I am still having this issue, appreciate if someone can help. On looking closely the FTP.connect() the connection to server is not a problem but getting acknowledgement (or the welcome message) from server is an issue. lftp does not have this issue and FileZilla does not have any issue either as in the log here -

Status: Connecting to xx.xx.xx.xxx:990...  
Status: Connection established, initializing TLS...  
Status: Verifying certificate...  
Status: TLS/SSL connection established, waiting for welcome message...  
Response:   220-      Vous allez vous connecter sur un serveur prive  
Response:   220-     Seules les personnes habilitees y sont autorisees  
Response:   220 Les contrevenants s'exposent aux poursuites prevues par la loi.  
Command:    USER xxxxxxxxxxxxx  
Response:   331 Password required for xxxxxxxxxxxxx.  
Command:    PASS **********  
Response:   230 Login OK. Proceed.  
Command:    PBSZ 0  
Response:   200 PBSZ Command OK. Protection buffer size set to 0.  
Command:    PROT P  
Response:   200 PROT Command OK. Using Private data connection  
Status: Connected  
Status: Retrieving directory listing...  
Command:    PWD  
Response:   257 "/" is current folder.  
Command:    TYPE I  
Response:   200 Type set to I.  
Command:    PASV  
Response:   227 Entering Passive Mode (81,93,20,199,4,206).  
Command:    MLSD  
Response:   150 Opening BINARY mode data connection for MLSD /.  
Response:   226 Transfer complete. 0 bytes transferred. 0 bps.  
Status: Directory listing successful  

Answer 1

I've worked on the same problem for half a day and finally figured it out.

For the implicit FTP TLS/SSL(defualt port 990), our client program must build a TLS/SSL connection right after the socket is created. But python's class FTP_TLS doesn't reload the connect function from class FTP. We need to fix it:

class tyFTP(ftplib.FTP_TLS):
  def __init__(self,
               host='',
               user='',
               passwd='',
               acct='',
               keyfile=None,
               certfile=None,
               timeout=60):

    ftplib.FTP_TLS.__init__(self,
                            host=host,
                            user=user,
                            passwd=passwd,
                            acct=acct,
                            keyfile=keyfile,
                            certfile=certfile,
                            timeout=timeout)

  def connect(self, host='', port=0, timeout=-999):
    """Connect to host.  Arguments are:
    - host: hostname to connect to (string, default previous host)
    - port: port to connect to (integer, default previous port)
    """
    if host != '':
        self.host = host
    if port > 0:
        self.port = port
    if timeout != -999:
        self.timeout = timeout
    try:
        self.sock = socket.create_connection((self.host, self.port), self.timeout)
        self.af = self.sock.family
        # add this line!!!
        self.sock = ssl.wrap_socket(self.sock,
                                    self.keyfile,
                                    self.certfile,
                                    ssl_version=ssl.PROTOCOL_TLSv1)
        # add end
        self.file = self.sock.makefile('rb')
        self.welcome = self.getresp()
    except Exception as e:
        print(e)
    return self.welcome

This derived class reloads the connect function and builds a wrapper around the socket to TLS. After you successfully connect and login to FTP server, you need to call: FTP_TLS.prot_p() before executing any FTP command!

Hope this will help ^_^

Answer 2

Extending upon NERV's response - which helped me immensely, here is how I was able to solve my problem with a implicit TLS connection on port 990 requiring authentication.

Filename: ImplicitTLS.py

from ftplib import FTP_TLS
import socket
import ssl

class tyFTP(FTP_TLS):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None, certfile=None, timeout=60):
        FTP_TLS.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)
    def connect(self, host='', port=0, timeout=-999):
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout

        try: 
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ssl_version=ssl.PROTOCOL_TLSv1)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print e
        return self.welcome

Then from my main application I did this:

from ImplicityTLS import tyFTP
server = tyFTP()
server.connect(host="xxxxx", port=990)
server.login(user="yyyy", passwd="fffff")
server.prot_p()

And that was it, i was able to download files, etc. Props go to NERV for the original answer.

Answer 3

I know this thread is quite old and ftp is not as popular as it once was but anyways, in case it helps anybody I'd like to make an additional contribution. I ran into a similar situation trying to connect to ftp server using IMPLICIT (Port 990) ftps in PASSIVE mode. In that situation the server, after the initial connection is negotiated, usually provides a new host IP address and port, likely different from the ones that were used to make the initial connection, over which the actual data transfers are supposed to happen. No big deal, ftps clients, including python, can handle this, only this particular server was providing a non-routable (likely internal to the firewall) IP address. I noticed that FileZilla connected no problem, but python ftplib could not. Then I ran into this thread:

How to replace a non routable IP address with server address on ftplib

which clued me in. Using Grzegorz Wierzowiecki methodology I extended the method alluded to in that thread and came up with this, which solved my problem.

import ftplib, os, sys
import socket
import ssl
FTPS_OBJ = ftplib.FTP_TLS



def conn_i_ftps(FTP_Site, Login_Name, Login_Password):
    print "Starting IMPLICIT ftp_tls..."
    ftps = tyFTP()
    print ftps.connect(host=FTP_Site, port=990, timeout=120)
    ftps.prot_p()
    ftps.login(user=Login_Name, passwd=Login_Password)
    print "Logged In"
    ftps.retrlines('LIST')
    # return ftps


class tyFTP(FTPS_OBJ):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None, certfile=None, timeout=60):
        FTPS_OBJ.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)

    def connect(self, host='', port=0, timeout=-999):
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout

        try:
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print e
        return self.welcome

    def makepasv(self):
        print port #<---Show passively assigned port
        print host #<---Show the non-routable, passively assigned IP
        host, port = FTPS_OBJ.makepasv(self)
        host = socket.gethostbyname(self.host) #<---- This changes the host back to the original IP that was used for the connection
        print 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
        print host #<----Showing the original IP
        return host, port

Then the code gets called like this

FTP_Site       =  "ftp.someserver.com"
Login_Name     =  "some_name"
Login_Password =  "some_passwd"

conn_i_ftps(FTP_Site, Login_Name, Login_Password)

I suppose one could wrap the lines where the host gets changed back with an IF statement identifying non-routable addresses, sort of like this:

if host.split(".")[0] in (10, 192, 172):
     host = socket.gethostbyname(self.host)
     .
     .
     .

Answer 4

Extending the solutions that have been proposed so far, the issue is that implicit FTPS connections need the socket to be ssl wrapped automatically, before we get a chance to call login(). A lot of the subclasses that people are proposing do this in the context of the connect method, we can more generally manage this by modifying the get/set of self.sock with a property to auto-wrap on set:

import ftplib
import ssl

class ImplicitFTP_TLS(ftplib.FTP_TLS):
    """FTP_TLS subclass that automatically wraps sockets in SSL to support implicit FTPS."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._sock = None

    @property
    def sock(self):
        """Return the socket."""
        return self._sock

    @sock.setter
    def sock(self, value):
        """When modifying the socket, ensure that it is ssl wrapped."""
        if value is not None and not isinstance(value, ssl.SSLSocket):
            value = self.context.wrap_socket(value)
        self._sock = value

Usage is essentially the same as with the standard FTP_TLS class:

ftp_client = ImplicitFTP_TLS()
ftp_client.connect(host='ftp.example.com', port=990)
ftp_client.login(user='USERNAME', passwd='PASSWORD')
ftp_client.prot_p()

Answer 5

The answer by NERV and the sample by Brad Decker was really helpful. Kudos to them. They saved me hours.

Unfortunetely, initially it didn't work for me.

In my case, the connection just worked once I removed the ssl_version parameter from the ssl.wrap_socket method. Also, to send any command to the server, I had to overwrite the ntransfercmd method from the FTP_TLS class and remove the ssl_version parameter there too.

That's the code that worked for me:

from ftplib import FTP_TLS, FTP
import socket
import ssl

class IMPLICIT_FTP_TLS(FTP_TLS):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None,
        certfile=None, timeout=60):
        FTP_TLS.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)

    def connect(self, host='', port=0, timeout=-999):
        '''Connect to host.  Arguments are:
        - host: hostname to connect to (string, default previous host)
        - port: port to connect to (integer, default previous port)
        '''
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout
        try:
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print (e)
        return self.welcome

    def ntransfercmd(self, cmd, rest=None):
        conn, size = FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = ssl.wrap_socket(conn, self.keyfile, self.certfile)
        return conn, size

And the obligatory sample:

>>> ftps = IMPLICIT_FTP_TLS()
>>> ftps.connect(host='your.ftp.host', port=990)
>>> ftps.login(user="your_user", passwd="your_passwd")
>>> ftps.prot_p()
>>> ftps.retrlines('LIST')