Is there a log file analyzer for log4j files?

Question

I am looking for some kind of analyzer tool for log files generated by log4j files. I am looking something more advanced than grep? What are you using for log file analysis?

I am looking for following kinds of features:

• The tool should tell me how many time a given log statement or a stack trace has occurred, preferably with support for some kinds of patterns (eg. number of log statements matching 'User [a-z]* logged in').
• Breakdowns by log level (how many INFO, DEBUG lines) and by class that initiated the log message would be nice.
• Breakdown by date (how many log statements in given time period)
• What log lines occur commonly together?
• Support for several files since I am using log rolling
• Hot spot analysis: find if there is a some time period when there is unusually high number of log statements
• Either command-line or GUI are fine
• Open Source is preferred but I am also interested in commercial offerings

My log4j configuration uses org.apache.log4j.PatternLayout with pattern %d %p %c - %m%n but that could be adapted for analyzer tool.

Answer 1

I have created a custom tool for that: https://plus.google.com/u/0/102275357970232913798/posts/Fsu6qftH2ja

Alfa is a GUI tool for analizing log files. Usually you are forced to search for data in them using editors. You open a log, press Ctrl-F and the "Next" button again and again, then reload the file as it was modified, and repeat the search. Alfa maps a log file to a database allowing you to use standard SQL queries to get data without any superfluous actions.

Answer 2

I'd suggest Splunk. It provides fast, Google-like searching across lots (terabytes) of logs, is easy to filter (e.g. by log level or date), makes it easy to correlate into transactions of multiple related log events, etc.

There's a downloadable version that's free as long as you're indexing less than 500MB of logs per day.