Verify whether ftp is complete or not?

Question

I got an application which is polling on a folder continuously. Once any file is ftp to the folder, the application has to move this file to some other folder for processing.

Here, we don't have any option to verify whether ftp is complete or not.

One command "lsof" is suggested in the technical forums. It got a file description column which gives the file status.

Since, this is a free bsd command and not present in old versions of linux, I want to clarify the usage of this command.

Can you guys tell us your experience in file verification and is there any other alternative solution available?

Also, is there any risk in using this utility?

Appreciate your help in advance.

Thanks, Mathew Liju

Answer 1

There are normally three approaches to this sort of problem.

1. providing a signal file so that when your file is transferred, an additional file is sent to mark that transfer is complete
2. add an entry to a log file within that directory to indicate a transfer is complete (this really only works if you have a single peer updating the directory, to avoid concurrency issues)
3. parsing the file to determine completeness. e.g. does the file start with a length field, or is it obviously incomplete ? e.g. parsing an incomplete XML file will result in a parse error due to the lack of an end element. Depending on your file's size and format, this can be trivial, or can be very time-consuming.

lsof would possibly be an option, although you've identified your Linux portability issue. If you use this, note the -F option, which formats the output suitable for processing by other programs, rather than being human-readable.

EDIT: Pax identified a fourth (!) method I'd forgotten - using the fact that the timestamp of the file hasn't updated in some time.

Answer 2

There is a fifth method. You can also check if the FTP Session is still active. This will work if every peer has it's own ftp user account. As long as the user is not logged off from FTP, assume the files are not complete.

Answer 3

We've done this before in a number of different ways.

Method one:

If you can control the process sending the files, have it send the file itself followed by a sentinel file. For example, send the real file "contracts.doc" followed by a one-byte "contracts.doc.sentinel".

Then have your listener process watch out for the sentinel files. When one of them is created, you should process the equivalent data file, then delete both.

Any data file that's more than a day old and doesn't have a corresponding sentinel file, get rid of it - it was a failed transmission.

Method two:

Keep an eye on the files themselves (specifically the last modification date/time). Only process files whose modification time is more than N minutes in the past. That increases the latency of processing the files but you can usually be certain that, if a file hasn't been written to in five minutes (for example), it's done.

Conclusion:

Both those methods have been used by us successfully in the past. I prefer the first but we had to use the second one once when we were not allowed to change the process sending the files.

The advantage of the first one is that you know the file is ready when the sentinel file appears. With both lsof (I'm assuming you're treating files that aren't open by any process as ready for processing) and the timestamps, it's possible that the FTP crashed in the middle and you may be processing half a file.