SFTP file manipulation using shell script

Question

Has anyone tried to access and manipulate files in an SFTP server using shell script Basically, here's what I have to do: 1. Open SFTP, access the designated directory 2. Loop through all .txt files 3. Read filename of each .txt files and only get the file/s that contain/s 'XX' substring (i.e. hello-XX.txt) 4. Rename that file by appending an '-OK' string (i.e.hello-XX-OK.txt)

Thanks for your inputs.

Answer 1

Nothing in your description requires you to run any logic on the sftp server. You can just get the data using the sftp command and do the processing locally, e.g.:

for file in $(echo 'ls -1' | sftp blah | tail -n +2); do
  # do stuff
done

Answer 2

You can script SFTP with a "here document" (<<EOF, as specified here), but you won't be able to use shell-script-like primitives (a la bash) in an SFTP script.

Perhaps try an SSH-based script instead, if you have SSH access.

Answer 3

Here's a starting point that I got working for my own purposes. I'm using sshpass so that the password can be specified but you should be able to remove that extra goo if you have the RSA public keys properly exchanged.

#!/bin/bash

function sftpexec()
{
    (SSHPASS="password" sshpass -e sftp user@server | tail -n +2) << !
$1
!
}

FILES=$(sftpexec ls)
echo $FILES

What this will do is list all the files on the server in the starting directory.

Answer 4

This is what expect is for. Roughly:

#!/PATH/TO/expect -f
spawn sftp host
expect -re "Username:"
send -- "MYUSER\r"
expect -re "Password:"
send -- "PASSWORD\r"
expect -re "ftp>"
send -- "cd mydir\r"
expect -re "ftp>"
send -- "get myfile\r"
...

Answer 5

Let's use Perl for that:

#!/usr/bin/perl
use Net::SFTP::Foreign;
$sftp = Net::SFTP::Foreign->new($host, user => $user, autodie => 1);
$sftp->setcwd($dir);
$sftp->mget("*XX*.txt", $local_dir);