Viewing large log files in JavaFX in a ListView

Question

I have written a JavaFx App to view log files. The files are opened each in a new tab and I run through the text and add it to the list views. I have created a cellfactory to highlight certain text in the list file and that is all working.

The problem is that the log files are between 800-1300Mb. Opening 3-4 at once and the JVM reached its memory limit.

Is there another way to do this to only have the text currently being displayed in the list view in memory and as I scroll load more into memory?

---

I need to be able to scroll using the scroll bars. I have search Google and I have found plenty of the same question. Where people want to display a large text 1-10Gb files in Java (In Swing) dated back a few years ago and there is no answer:

• http://www.dreamincode.net/forums/topic/224430-reading-in-large-text-files1gb-10gb-java-swing/
• http://www.javaprogrammingforums.com/awt-java-swing/9577-diplay-large-text-jtextarea-2gb.html
• http://www.dreamincode.net/forums/topic/224430-reading-in-large-text-files1gb-10gb-java-swing/

I had a look at pagination but I need to be able to use the scroll bars and I need it seemlesly. The user should not know that certain portions is not in memory. They scroll and it display the next lines.

Have anyone ever found a solution to this in Java (Swing or JavaFX)?

Answer 1

A couple of possible approaches to prevent memory issues viewing large log files:

1. Use the Pagination control and load parts of a log file at a time OR

2. Subclass ObservableList and back it by something like a Guava Iterable that loads data on demand.

Both approaches load only log file data that is needed for the current view into memory rather than loading the entire log file into memory.

Answer 2

The key is to forget about line numbers. Unless each line is exactly the same number of bytes long, line numbers are an abstraction that requires you to read the whole file to find the position of the next line.

Instead think byte position, since you can seek to that without reading the bytes between, by using a RandomAccessFile or a SeekableByteChannel. Of course, then you have to return the byte position to the client app so you can return it to the server when you are getting the next page.

Using the scrollbar to do paging is a neat and intuitive trick. The hard part is re-writing the scrollbar to show the whole file instead of just what you have in memory.

NOTE: Just noticed how old this question is... but hopefully my comments are useful for the next searcher.