Tailing Log File and Write results to new file

Question

I'm not sure how to word this so I'll type it out and then edit and answer any questions that come up..

Currently on my local network device (PHP4 based) I'm using this to tail a live system log file: http://commavee.com/2007/04/13/ajax-logfile-tailer-viewer/

This works well and every 1 second it loads an external page (logfile.php) that does a tail -n 100 logfile.log The script doesn't do any buffering so the results it displayes onscreen are the last 100 lines from the log file.

The logfile.php contains :

<? // logtail.php $cmd = "tail -10 /path/to/your/logs/some.log"; exec("$cmd 2>&1", $output);
foreach($output as $outputline) {
 echo ("$outputline\n");
}
?>

This part is working well.

I have adapted the logfile.php page to write the $outputline to a new text file, simply using fwrite($fp,$outputline."\n");

Whilst this works I am having issues with duplication in the new file that is created.

Obviously each time tail -n 100 is run produces results, the next time it runs it could produce some of the same lines, as this repeats I can end up with multiple lines of duplication in the new text file.

I can't directly compare the line I'm about to write to previous lines as there could be identical matches.

Is there any way I can compare this current block of 100 lines with the previous block and then only write the lines that are not matching.. Again possible issue that block A & B will contain identical lines that are needed...

Is it possible to update logfile.php to note the position it last tooked at in my logfile and then only read the next 100 lines from there and write those to the new file ?

The log file could be upto 500MB so I don't want to read it all in each time..

Any advice or suggestions welcome..

Thanks

UPDATE @ 16:30

I've sort of got this working using :

$file = "/logs/syst.log";
$handle = fopen($file, "r");

if(isset($_SESSION['ftell'])) {   
    clearstatcache();
    fseek($handle, $_SESSION['ftell']); 

    while ($buffer = fgets($handle)) { 
        echo $buffer."<br/>";
        @ob_flush(); @flush();
    }   

    fclose($handle);
    @$_SESSION['ftell'] = ftell($handle);        
} else {
    fseek($handle, -1024, SEEK_END);
    fclose($handle);
     @$_SESSION['ftell'] = ftell($handle);
}

This seems to work, but it loads the entire file first and then just the updates.

How would I get it start with the last 50 lines and then just the updates ?

Thanks :)

UPDATE 04/06/2013 Whilst this works it's very slow with large files.

I've tried this code and it seems faster, but it doesn't just read from where it left off.

function last_lines($path, $line_count, $block_size = 512){
    $lines = array();

    // we will always have a fragment of a non-complete line
    // keep this in here till we have our next entire line.
    $leftover = "";

    $fh = fopen($path, 'r');
    // go to the end of the file
    fseek($fh, 0, SEEK_END);
    do{
        // need to know whether we can actually go back
        // $block_size bytes
        $can_read = $block_size;
        if(ftell($fh) < $block_size){
            $can_read = ftell($fh);
        }

        // go back as many bytes as we can
        // read them to $data and then move the file pointer
        // back to where we were.
        fseek($fh, -$can_read, SEEK_CUR);
        $data = fread($fh, $can_read);
        $data .= $leftover;
        fseek($fh, -$can_read, SEEK_CUR);

        // split lines by \n. Then reverse them,
        // now the last line is most likely not a complete
        // line which is why we do not directly add it, but
        // append it to the data read the next time.
        $split_data = array_reverse(explode("\n", $data));
        $new_lines = array_slice($split_data, 0, -1);
        $lines = array_merge($lines, $new_lines);
        $leftover = $split_data[count($split_data) - 1];
    }
    while(count($lines) < $line_count && ftell($fh) != 0);
    if(ftell($fh) == 0){
        $lines[] = $leftover;
    }
    fclose($fh);
    // Usually, we will read too many lines, correct that here.
    return array_slice($lines, 0, $line_count);
}

Any way this can be amend so it will read from the last known position.. ?

Thanks

Answer 1

Introduction

You can tail a file by tracking the last position;

Example

$file = __DIR__ . "/a.log";
$tail = new TailLog($file);
$data = $tail->tail(100) ;
// Save $data to new file 

TailLog is a simple class i wrote for this task here is a simple example to show its actually tailing the file

Simple Test

$file = __DIR__ . "/a.log";
$tail = new TailLog($file);

// Some Random Data
$data = array_chunk(range("a", "z"), 3);

// Write Log
file_put_contents($file, implode("\n", array_shift($data)));

// First Tail (2) Run
print_r($tail->tail(2));

// Run Tail (2) Again
print_r($tail->tail(2));

// Write Another data to Log
file_put_contents($file, "\n" . implode("\n", array_shift($data)), FILE_APPEND);

// Call Tail Again after writing Data
print_r($tail->tail(2));

// See the full content
print_r(file_get_contents($file));

Output

// First Tail (2) Run
Array
(
    [0] => c
    [1] => b
)

// Run Tail (2) Again
Array
(
)

// Call Tail Again after writing Data
Array
(
    [0] => f
    [1] => e
)

// See the full content
a
b
c
d
e
f

Real Time Tailing

while(true) {
    $data = $tail->tail(100);
    // write data to another file
    sleep(5);
}

Note: Tailing 100 lines does not mean it would always return 100 lines. It would return new lines added 100 is just the maximum number of lines to return. This might not be efficient where you have heavy logging of more than 100 line per sec is there is any

Tail Class

class TailLog {
    private $file;
    private $data;
    private $timeout = 5;
    private $lock;

    function __construct($file) {
        $this->file = $file;
        $this->lock = new TailLock($file);
    }

    public function tail($lines) {
        $pos = - 2;
        $t = $lines;
        $fp = fopen($this->file, "r");
        $break = false;
        $line = "";
        $text = array();

        while($t > 0) {
            $c = "";

            // Seach for End of line
            while($c != "\n" && $c != PHP_EOL) {
                if (fseek($fp, $pos, SEEK_END) == - 1) {
                    $break = true;
                    break;
                }
                if (ftell($fp) < $this->lock->getPosition()) {
                    break;
                }
                $c = fgetc($fp);
                $pos --;
            }
            if (ftell($fp) < $this->lock->getPosition()) {
                break;
            }
            $t --;
            $break && rewind($fp);
            $text[$lines - $t - 1] = fgets($fp);
            if ($break) {
                break;
            }
        }

        // Move to end
        fseek($fp, 0, SEEK_END);

        // Save Position
        $this->lock->save(ftell($fp));

        // Close File
        fclose($fp);
        return array_map("trim", $text);
    }
}

Tail Lock

class TailLock {
    private $file;
    private $lock;
    private $data;

    function __construct($file) {
        $this->file = $file;
        $this->lock = $file . ".tail";
        touch($this->lock);

        if (! is_file($this->lock))
            throw new Exception("can't Create Lock File");

        $this->data = json_decode(file_get_contents($this->lock));

        // Check if file is valida json
        // Check if Data in the original files as not be delete
        // You expect data to increate not decrease

        if (! $this->data || $this->data->size > filesize($this->file)) {
            $this->reset($file);
        }
    }

    function getPosition() {
        return $this->data->position;
    }

    function reset() {
        $this->data = new stdClass();
        $this->data->size = filesize($this->file);
        $this->data->modification = filemtime($this->file);
        $this->data->position = 0;
        $this->update();
    }

    function save($pos) {
        $this->data = new stdClass();
        $this->data->size = filesize($this->file);
        $this->data->modification = filemtime($this->file);
        $this->data->position = $pos;
        $this->update();
    }

    function update() {
        return file_put_contents($this->lock, json_encode($this->data, 128));
    }
}

Answer 2

Use tail -c <number of bytes, instead of number of lines, and then check the file size. The rough idea is:

$old_file_size = 0;
$max_bytes = 512;

function last_lines($path) {
  $new_file_size = filesize($path);
  $pending_bytes = $new_file_size - $old_file_size;
  if ($pending_bytes > $max_bytes) $pending_bytes = $max_bytes;
  exec("tail -c " + $pending_bytes + " /path/to/your_log", $output);
  $old_file_size = $new_file_size;
  return $output;
}

The advantage is that you can do away with all the special processing stuff, and get good performance. The disadvantage is that you have to manually split the output into lines, and probably you could end up with unfinished lines. But this isn't a big deal, you can easily work around by omitting the last line alone from the output (and appropriately subtracting the last line number of bytes from old_file_size).

Answer 3

Not really clear on how you want to use the output but would something like this work ....

$dat = file_get_contents("tracker.dat");
$fp = fopen("/logs/syst.log", "r");
fseek($fp, $dat, SEEK_SET);
ob_start();
// alternatively you can do a while fgets if you want to interpret the file or do something
fpassthru($fp);
$pos = ftell($fp);
fclose($fp);
echo nl2br(ob_get_clean());
file_put_contents("tracker.dat", ftell($fp));

tracker.dat is just a text file that contains where the read position position was from the previous run. I'm just seeking to that position and piping the rest to the output buffer.