copy files from FTP to HDFS

Question

I need to copy files from FTP sever outside my cluster to Hadoop i.e HDFS.

Thanks in advance.

Answer 1

Have you tried this? FTP TO HDFS ... You can modify this code and change arguments for File to be FTP a agrs[0] and HDFS Paht to args[1]. and than run it as hadoop jar. hope this'll help...

Answer 2

Try this:

hadoop fs -get ftp://uid:password@server_url/file_path temp_file | hadoop fs -moveFromLocal tmp_file hadoop_path/dest_file

Answer 3

Have you looked at WebHDFS (http://hadoop.apache.org/docs/r1.0.4/webhdfs.html) or HttpFS (http://hadoop.apache.org/docs/r2.2.0/hadoop-hdfs-httpfs/index.html)

The services need access to the Hadoop cluster, then you could expose the HttpFS port to a server with access to the FTP server.