How do you get last some lines of file via SFTP in

问题:

I need to login to a production server retrieve a file and update my data base with the data in this file. Since this is a production database, I don't want to get the whole file every 5 minutes since the file may be huge and this may impact the server. I need to get the last 30 lines of this file every 5 minutes interval and have as little impact as possible.

The following is my current code, I would appreciate any insight to how best accomplish this:

<?php

$user="id";
$pass="passed";
$c = curl_init("sftp://$user:$[e-mail]/opt/vmstat_server1");
curl_setopt($c, CURLOPT_PROTOCOLS, CURLPROTO_SFTP);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($c);
curl_close($c);
$data = explode("\n", $data);
?>

回答1:

SFTP is not capable of partial file transfers. You might have better luck using a fullblowin SSH connection and use a remote 'tail' operation to get the last lines of the file, e.g.

$lines = shell_exec("ssh [e-mail] 'tail -30 the_file'");

Of course, you might want to have something a little more robust that can handle things like net.glitches that prevent ssh from getting through, but as a basic starting point, this should do the trick.

回答2:

Marc B is wrong. SFTP is perfectly capable of partial file transfers. Here's an example of how to do what you want with phpseclib, a pure PHP SFTP implementation:

<?php
include('Net/SFTP.php');

$sftp = new Net_SFTP('www.domain.tld');
if (!$sftp->login('username', 'password')) {
    exit('Login Failed');
}

$size = $sftp->size('filename.remote');

// outputs the last ten bytes of filename.remote
echo $sftp->get('filename.remote', false, $size - 10);
?>

In fact I'd recommend an approach like this anyway since some SFTP servers don't let you run commands via the system shell. Plus, SFTP can work on Windows SFTP servers whereas tail is unlikely to do so even if you do have shell access. ie. overall, it's a lot more portable a solution.

If you want to get the last x lines of a file you could loop repeatedly, reading however many bytes each time, until you encounter 10x new line characters. ie. get the last 10 bytes, then the next to last 10 bytes, then the ten bytes before those ten bytes, etc.

回答3:

An answer by @Sammitch to a duplicate question Get last 15 lines from a large file in SFTP with phpseclib:

The following should result in a blob of text with at least 15 lines from the end of the file that you can then process further with your existing logic. You may want to tweak some of the logic depending on if your file ends with a trailing newline, etc.

$filename = './file.txt'

$filesize = $sftp->size($filename);
$buffersize = 4096;

$offset = $filesize; // start at the end
$result = '';
$lines = 0;

while( $offset > 0 && $lines < 15 ) {
  // work backwards
  if( $offset < $buffersize ) {
    $offset = 0;
  } else {
    $offset -= $buffer_size;
  }
  $buffer = $sftp->get($filename, false, $offset, $buffer_size));
  // count the number of newlines as we go
  $lines += substr_count($buffer, "\n");
  $result = $buffer . $result;
}