Ignore missing file while downloading with Python

Question

I am trying to download a certain file (named 010010-99999-year.gz) from an FTP server. This same file, but for different years is residing in different FTP directories. For instance:

ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/2000/010010-99999-1973.gz ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/2001/010010-99999-1974.gz and so on. The picture illustrates one of the directories:

The file is not located in all the directories (i.e. all years). In such case I want the script to ignore that missing files, print "not available", and continue with the next directory (i.e. next year). I could do this using the NLST listing by first generating a list of files in the current FTP directory and then checking if my file is on that list, but that is slow, and NOAA (the organization owning the server) does not like file listing (source). Therefore I came up with this code:

def FtpDownloader2(url="ftp.ncdc.noaa.gov"):
    ftp=FTP(url)        
    ftp.login()
    for year in range(1901,2015):
        ftp.cwd("/pub/data/noaa/isd-lite")
        ftp.cwd(str(year))
        fullStationId="010010-99999-%s.gz" % year
        try:              
            file=open(fullStationId,"wb")
            ftp.retrbinary('RETR %s' % fullStationId, file.write)
            print("File is available")
            file.close()
        except: 
            print("File not available")
    ftp.close()

This downloads the existing files (year 1973-2014) correctly, but it is also generating empty files for years 1901-1972. The file is not in the FTP for 1901-1972. Am I doing anything wrong in the use of try and except, or is it some other issue?

Answer 1

I took your code and modified it a little:

from ftplib import FTP, error_perm
import os

def FtpDownloader2(url="ftp.ncdc.noaa.gov"):
    ftp = FTP(url)
    ftp.login()
    for year in range(1901, 2015):
        remote_file = '/pub/data/noaa/isd-lite/{0}/010010-99999-{0}.gz'.format(year)
        local_file = os.path.basename(remote_file)
        try:
            with open(local_file, "wb") as file_handle:
                ftp.retrbinary('RETR %s' % remote_file, file_handle.write)
            print('OK', local_file)
        except error_perm:
            print('ERR', local_file)
            os.unlink(local_file)
    ftp.close()

Notes

• The most dangerous operation a person can do is to have an except clause without a specific exception class. This type of construct will ignore all errors, making it hard to troubleshoot. To fix this, I added the specific exception error_perm
• Once the exception occurred, I absolutely know for sure that the local file is closed because the with statement guarantees that
• I removed the local file if error_perm exception occurred, a sign that the file is not available from the server
• I removed the code to change directories: for each year, you cwd twice which slows down the process
• range(1901, 2015) will not include 2015. If you want it, you have to specify range(1901, 2016)
• I improved the print statements to include the file names, making it easier to track which ones are available and which ones are not

Update

This update answers your question regarding not creating empty local file (then having to delete them). There are a couple of different ways:

1. Query the remote file's existence before downloading. Only create the local file when the remote exists. The problem with this approach is querying a remote file takes longer than creating/deleting a local file.
2. Create a string buffer (StringIO), download to that buffer. Only create a local file when that string buffer is not empty. The problem with this approach is you are writing the same data twice: once to the string buffer, and once from the string buffer to the file.

Answer 2

I think the problem is within your try: except block, where you keep a file handler open for a new file before checking if the file exists or not:

try:              
    file=open(fullStationId,"wb")
    ftp.retrbinary('RETR %s' % fullStationId, file.write)
    print("File is available")
    file.close()
except: 
    print("File not available")

Instead, add an additional statement in the except block to close the file handler, and another statement to remove the file if it is empty.

Another possibility is to open the file for writing locally only if the file exists and has a non zero size on the server using ftp.size