How do I parse a listing of files to get just the

Question

So lets say I'm using Python's ftplib to retrieve a list of log files from an FTP server. How would I parse that list of files to get just the file names (the last column) inside a list? See the link above for example output.

Answer 1

Using retrlines() probably isn't the best idea there, since it just prints to the console and so you'd have to do tricky things to even get at that output. A likely better bet would be to use the nlst() method, which returns exactly what you want: a list of the file names.

Answer 2

Since every filename in the output starts at the same column, all you have to do is get the position of the dot on the first line:

drwxrwsr-x 5 ftp-usr pdmaint 1536 Mar 20 09:48 .

Then slice the filename out of the other lines using the position of that dot as the starting index.

Since the dot is the last character on the line, you can use the length of the line minus 1 as the index. So the final code is something like this:

lines = ftp.retrlines('LIST')
lines = lines.split("\n") # This should split the string into an array of lines

filename_index = len(lines[0]) - 1
files = []

for line in lines:
    files.append(line[filename_index:])

Answer 3

And a slightly less-optimal method, by the way, if you're stuck using retrlines() for some reason, is to pass a function as the second argument to retrlines(); it'll be called for each item in the list. So something like this (assuming you have an FTP object named 'ftp') would work as well:

filenames = []
ftp.retrlines('LIST', lambda line: filenames.append(line.split()[-1]))

The list 'filenames' will then be a list of the file names.

Answer 4

Is there any reason why ftplib.FTP.nlst() won't work for you? I just checked and it returns only names of the files in a given directory.

Answer 5

This best answer

You may want to use ftp.nlst() instead of ftp.retrlines(). It will give you exactly what you want.

If you can't, read the following :

Generators for sysadmin processes

In his now famous review, Generator Tricks For Systems Programmers An Introduction, David M. Beazley gives a lot of receipes to answer to this kind of data problem with wuick and reusable code.

E.G :

# empty list that will receive all the log entry
log = [] 
# we pass a callback function bypass the print_line that would be called by retrlines
# we do that only because we cannot use something better than retrlines
ftp.retrlines('LIST', callback=log.append)
# we use rsplit because it more efficient in our case if we have a big file
files = (line.rsplit(None, 1)[1] for line in log)
# get you file list
files_list = list(files)

Why don't we generate immediately the list ?

Well, it's because doing it this way offer you much flexibility : you can apply any intermediate generator to filter files before turning it into files_list : it's just like pipe, add a line, you add a process without overheat (since it's generators). And if you get rid off retrlines, it still work be it's even better because you don't store the list even one time.

EDIT : well, I read the comment to the other answer and it says that this won't work if there is any space in the name.

Cool, this will illustrate why this method is handy. If you want to change something in the process, you just change a line. Swap :

files = (line.rsplit(None, 1)[1] for line in log)

and

# join split the line, get all the item from the field 8 then join them
files = (' '.join(line.split()[8:]) for line in log)

Ok, this may no be obvious here, but for huge batch process scripts, it's nice :-)

Answer 6

If the FTP server supports the MLSD command, then please see section “single directory case” from that answer.

Use an instance (say ftpd) of the FTPDirectory class, call its .getdata method with connected ftplib.FTP instance in the correct folder, then you can:

directory_filenames= [ftpfile.name for ftpfile in ftpd.files]