How to parse using Grok from Java.. Is there any e

Question

I have seen Grok being very strong and lethal in parsing the log data. I wanted to use Grok for log parsing in our application, which is in java.. How can i connect/work with Grok from Java.?

Answer 1

Check out this Java library

https://github.com/aicer/grok

You can include it in your project as a maven dependency

<dependency>
    <groupId>org.aicer.grok</groupId>
    <artifactId>grok</artifactId>
    <version>0.9.0</version>
</dependency>

It comes with pre-defined patterns and you can also add yours.

The named patterns are extracted and the results are available in a map with the groups names as the keys and the retrieved values are mapped to these keys.

final GrokDictionary dictionary = new GrokDictionary();

// Load the built-in dictionaries
dictionary.addBuiltInDictionaries();

// Add custom pattern
dictionary.addDictionary(new File(patternDirectoryOrFilePath));

// Resolve all expressions loaded
dictionary.bind();

This next examples adds string patterns directly to the dictionary without using a file

final GrokDictionary dictionary = new GrokDictionary();

// Load the built-in dictionaries
dictionary.addBuiltInDictionaries();

// Add custom pattern directly

dictionary.addDictionary(new StringReader("DOMAINTLD [a-zA-Z]+"));
dictionary.addDictionary(new StringReader("EMAIL %{NOTSPACE}@%{WORD}\.%{DOMAINTLD}"));

// Resolve all expressions loaded
dictionary.bind();

Here is a complete example of how to use the library

    public final class GrokStage {

  private static final void displayResults(final Map<String, String> results) {
    if (results != null) {
      for(Map.Entry<String, String> entry : results.entrySet()) {
        System.out.println(entry.getKey() + "=" + entry.getValue());
      }
    }
  }

  public static void main(String[] args) {

    final String rawDataLine1 = "1234567 - [e-mail] cc55ZZ35 1789 Hello Grok";
    final String rawDataLine2 = "98AA541 - [e-mail] mmddgg22 8800 Hello Grok";
    final String rawDataLine3 = "55BB778 - [e-mail] secret123 4439 Valid Data Stream";

    final String expression = "%{EMAIL:username} %{USERNAME:password} %{INT:yearOfBirth}";

    final GrokDictionary dictionary = new GrokDictionary();

    // Load the built-in dictionaries
    dictionary.addBuiltInDictionaries();

    // Resolve all expressions loaded
    dictionary.bind();

    // Take a look at how many expressions have been loaded
    System.out.println("Dictionary Size: " + dictionary.getDictionarySize());

    Grok compiledPattern = dictionary.compileExpression(expression);

    displayResults(compiledPattern.extractNamedGroups(rawDataLine1));
    displayResults(compiledPattern.extractNamedGroups(rawDataLine2));
    displayResults(compiledPattern.extractNamedGroups(rawDataLine3));
  }
}

Answer 2

Try downloading java-grok from GitHub: https://github.com/NFLabs/java-grok You can test patterns using the Grok Debugger: http://grokdebug.herokuapp.com/