Watch/read a growing log file

Question

I have a log file that is constantly growing. How can I watch and parse it via a Ruby script?

The script will parse each new line as it is written to the file and output something to the screen when the new line contains the string 'ERROR'

Answer 1

check out file-tail gem

Answer 2

Working on idea of @Qianjigui but not using 100% CPU:

def watch_for(file, pattern)
  # Replace -n0 with -n+1 if you want to read from the beginning of file
  f = IO.popen(%W[tail -f -n0 #{file}])
  loop do
    select([f])
    while line = f.gets
      puts "Found it! #{line}" if line =~ pattern
    end
  end
end

watch_for('g.txt', /ERROR/)

Answer 3

You can use Kernel#select in the following way:

def watch_for(file,pattern)
   f = File.open(file,"r")

   # Since this file exists and is growing, seek to the end of the most recent entry
   f.seek(0,IO::SEEK_END)

   while true
      select([f])
      puts "Found it!" if f.gets =~ pattern
   end
end

Then call it like:

watch_for("some_file", /ERROR/)

I've elided all error checking and such - you will want to have that and probably some mechanism to break out of the loop. But the basic idea is there.

Answer 4

Poor man's approach for quick stuff:

1. a Ruby script that does

   ARGF.each do |line|
     ...
   

2. Running screen with:

   tail -f file | ruby script 
   

Answer 5

def watch_for(file, pattern)
  f = File.open(file,"r")
  f.seek(0,IO::SEEK_END)
  while true do
    select([f])
    line = f.gets
    puts "Found it! #{line}" if line=~pattern
  end
end

watch_for("g.txt",/ERROR/)

Thanks for the ezpz's idea, using the select method you get get what you want. The select method is listening the IO's stream, read the bytes what comes 'late'.

Answer 6

There are two approach:

• poll the file in an infinite loop (like in Qianjigui's answer, but it is good to put some sleep inside the infinite loop)
• use OS event subsystem: kqueue on BSD, inotify on Linux

Here is an article I wrote about this: Ruby for Admins: Reading Growing Files. So the program combining both event subsystem and polling looks like this:

def tail_dash_f(filename)
  open(ARGV.first) do |file|
    file.read          
    case RUBY_PLATFORM   # string with OS name, like "amd64-freebsd8"
    when /bsd/, /darwin/
      require 'rb-kqueue'
      queue = KQueue::Queue.new     
      queue.watch_file(ARGV.first, :extend) do
        yield file.read             
      end
      queue.run                     
    when /linux/
      require 'rb-inotify'
      queue = INotify::Notifier.new  
      queue.watch(ARGV.first, :modify) do
        yield file.read             
      end
      queue.run                      
    else
      loop do           
        changes = file.read
        unless changes.empty?  
          yield changes
        end
        sleep 1.0       
      end
    end
  end
end

tail_dash_f ARGV.first do |data|
  print data
  if data =~ /error/i
    # do something else, for example send an email to administrator
  end
end