Read a file line by line in reverse order

Question

I have a java ee application where I use a servlet to print a log file created with log4j. When reading log files you are usually looking for the last log line and therefore the servlet would be much more useful if it printed the log file in reverse order. My actual code is:

    response.setContentType("text");
    PrintWriter out = response.getWriter();
    try {
        FileReader logReader = new FileReader("logfile.log");
        try {
            BufferedReader buffer = new BufferedReader(logReader);
            for (String line = buffer.readLine(); line != null; line = buffer.readLine()) {
                out.println(line);
            }
        } finally {
            logReader.close();
        }
    } finally {
        out.close();
    }

The implementations I've found in the internet involve using a StringBuffer and loading all the file before printing, isn't there a code light way of seeking to the end of the file and reading the content till the start of the file?

Answer 1

Concise solution using Java 7 Autoclosables and Java 8 Streams :

try (Stream<String> logStream = Files.lines(Paths.get("C:\\logfile.log"))) {
   logStream
      .sorted(Comparator.reverseOrder())
      .limit(10) // last 10 lines
      .forEach(System.out::println);
}

Big drawback: only works when lines are strictly in natural order, like log files prefixed with timestamps but without exceptions

Answer 2

Good question. I'm not aware of any common implementations of this. It's not trivial to do properly either, so be careful what you choose. It should deal with character set encoding and detection of different line break methods. Here's the implementation I have so far that works with ASCII and UTF-8 encoded files, including a test case for UTF-8. It does not work with UTF-16LE or UTF-16BE encoded files.

import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.RandomAccessFile;
import java.io.Reader;
import java.io.UnsupportedEncodingException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import junit.framework.TestCase;

public class ReverseLineReader {
    private static final int BUFFER_SIZE = 8192;

    private final FileChannel channel;
    private final String encoding;
    private long filePos;
    private ByteBuffer buf;
    private int bufPos;
    private byte lastLineBreak = '\n';
    private ByteArrayOutputStream baos = new ByteArrayOutputStream();

    public ReverseLineReader(File file, String encoding) throws IOException {
        RandomAccessFile raf = new RandomAccessFile(file, "r");
        channel = raf.getChannel();
        filePos = raf.length();
        this.encoding = encoding;
    }

    public String readLine() throws IOException {
        while (true) {
            if (bufPos < 0) {
                if (filePos == 0) {
                    if (baos == null) {
                        return null;
                    }
                    String line = bufToString();
                    baos = null;
                    return line;
                }

                long start = Math.max(filePos - BUFFER_SIZE, 0);
                long end = filePos;
                long len = end - start;

                buf = channel.map(FileChannel.MapMode.READ_ONLY, start, len);
                bufPos = (int) len;
                filePos = start;
            }

            while (bufPos-- > 0) {
                byte c = buf.get(bufPos);
                if (c == '\r' || c == '\n') {
                    if (c != lastLineBreak) {
                        lastLineBreak = c;
                        continue;
                    }
                    lastLineBreak = c;
                    return bufToString();
                }
                baos.write(c);
            }
        }
    }

    private String bufToString() throws UnsupportedEncodingException {
        if (baos.size() == 0) {
            return "";
        }

        byte[] bytes = baos.toByteArray();
        for (int i = 0; i < bytes.length / 2; i++) {
            byte t = bytes[i];
            bytes[i] = bytes[bytes.length - i - 1];
            bytes[bytes.length - i - 1] = t;
        }

        baos.reset();

        return new String(bytes, encoding);
    }

    public static void main(String[] args) throws IOException {
        File file = new File("my.log");
        ReverseLineReader reader = new ReverseLineReader(file, "UTF-8");
        String line;
        while ((line = reader.readLine()) != null) {
            System.out.println(line);
        }
    }

    public static class ReverseLineReaderTest extends TestCase {
        public void test() throws IOException {
            File file = new File("utf8test.log");
            String encoding = "UTF-8";

            FileInputStream fileIn = new FileInputStream(file);
            Reader fileReader = new InputStreamReader(fileIn, encoding);
            BufferedReader bufReader = new BufferedReader(fileReader);
            List<String> lines = new ArrayList<String>();
            String line;
            while ((line = bufReader.readLine()) != null) {
                lines.add(line);
            }
            Collections.reverse(lines);

            ReverseLineReader reader = new ReverseLineReader(file, encoding);
            int pos = 0;
            while ((line = reader.readLine()) != null) {
                assertEquals(lines.get(pos++), line);
            }

            assertEquals(lines.size(), pos);
        }
    }
}

Answer 3

A simpler alternative, because you say that you're creating a servlet to do this, is to use a LinkedList to hold the last N lines (where N might be a servlet parameter). When the list size exceeds N, you call removeFirst().

From a user experience perspective, this is probably the best solution. As you note, the most recent lines are the most important. Not being overwhelmed with information is also very important.

Answer 4

The simplest solution is to read through the file in forward order, using an ArrayList<Long> to hold the byte offset of each log record. You'll need to use something like Jakarta Commons CountingInputStream to retrieve the position of each record, and will need to carefully organize your buffers to ensure that it returns the proper values:

FileInputStream fis = // .. logfile
BufferedInputStream bis = new BufferedInputStream(fis);
CountingInputStream cis = new CountingInputSteam(bis);
InputStreamReader isr = new InputStreamReader(cis, "UTF-8");

And you probably won't be able to use a BufferedReader, because it will attempt to read-ahead and throw off the count (but reading a character at a time won't be a performance problem, because you're buffering lower in the stack).

To write the file, you iterate the list backwards and use a RandomAccessFile. There is a bit of a trick: to properly decode the bytes (assuming a multi-byte encoding), you will need to read the bytes corresponding to an entry, and then apply a decoding to it. The list, however, will give you the start and end position of the bytes.

One big benefit to this approach, versus simply printing the lines in reverse order, is that you won't damage multi-line log messages (such as exceptions).