Alternative IndexProvider for Neo4J 1.9.1

Question

I'm using Lucene 4 in my application and don't want to change this. I'm trying to integrate Neo4J which bundles Lucene 3.5 as an IndexProvider implementation, neo4j-lucene-index.

Unfortunately, neo4j-lucene-index does not work, and with that dependency excluded, the app just hangs indefinitely on start up. I've tried neo4j-lucene4-index but that does not seem to be maintained very well and needs to be updated quite significantly to work with Neo4J 1.9.1. The changes go way beyond my understanding of the internals of Neo4J.

However, I can see that IndexProviders are pluggable, so I'm hoping that there is an existing alternative to Lucene - I can't find it at the moment though. Can anyone point me in the right direction for one?

It seems strange that Lucene 4 has been out for so long now and Neo4J doesn't support it. Am I missing something?

Currently, my POM looks like this for my Neo4J config:

<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-neo4j</artifactId>
    <version>2.2.1.RELEASE</version>
    <exclusions>
        <exclusion>
        <artifactId>neo4j</artifactId>
        <groupId>org.neo4j</groupId>
        </exclusion>
        <exclusion>
        <artifactId>neo4j-cypher</artifactId>
        <groupId>org.neo4j</groupId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.neo4j</groupId>
    <artifactId>neo4j-kernel</artifactId>
    <version>1.9.1</version>
    <exclusion>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-lucene-index</artifactId>
    </exclusion>
</dependency>

<dependency>
    <groupId>org.neo4j.app</groupId>
    <artifactId>neo4j-server</artifactId>
    <version>1.9.1</version>
    <exclusions>
        <exclusion>
        <artifactId>neo4j</artifactId>
        <groupId>org.neo4j</groupId>
        </exclusion>
        <exclusion>
        <artifactId>neo4j-cypher</artifactId>
        <groupId>org.neo4j</groupId>
        </exclusion>
        <exclusion>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-lucene-index</artifactId>
        </exclusion>
    </exclusions>
</dependency>

    <!-- A temporary dependency until Neo4J builds in support for Lucene 4. 
    Looks like they're planning to incorporate this project anyway This project 
    is available on GitHub, and needs to be built with: mvn license:format mvn 
    install to install into your local repo. 
        <dependency>
            <groupId>com.keatext</groupId>
            <artifactId>neo4j-lucene4-index</artifactId>
            <version>1.9.M01-SNAPSHOT</version>
        </dependency>-->

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-validator</artifactId>
    <version>5.0.1.Final</version>
</dependency>

Answer 1

If you don't care what index Neo4j uses, and you're using Maven to manage dependencies, you can use the Maven Shade plugin's class relocation feature to rename Neo4j's Lucene dependency so it doesn't conflict with other dependencies on newer versions of Lucene.

In my case this required moving Neo4j-dependent code into a separate Maven project, as Shade acts on a whole project/jar at once. So if you can get your conflicting Lucene dependencies into different projects, this should work great.

Answer 2

Some time ago, I was also faced with this issue: I was working on prototyping, and I really liked embedded mode of Neo4j. But, once I decided to use Lucene 4 - I was stumbled with incompatibility issue.

OSGi

As been suggested here: How to use two versions of jar in my java project - one of the possible solution is to use OSGi, and wrap Neo4j and Lucene 4 into different bundles. Each bundle will have separate classloader - so Neo4j will use in runtime classes from Lucene 3, but you still be able to use Lucene 4 for your purposes.

But, as far as I was working on prototyping - I didn't want to spend time on adaptation of my project for OSGi platform by the single reason of incompatibility of two components.

Maven Shade Plugin

So, I have solved issue with help of Maven Shade Plugin.

Maven Shade Plugin provides an ability to merge all dependencies into single "fat" JAR (also, called "uber JAR").

So, you can generate "uber Neo4j dependency", and use it in your project - instead of "real" Neo4j dependency.

But there is an additional important moment: Lucene 3 and Lucene 4 has the same package structure, and many classes still have the same names. So, this might cause classloading conflicts.

To address this issue, Maven Shade Plugin provides an ability to relocate classes during generation of of "uber JAR": http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

You can specify package name, and during packaging - Shade Plugin will move classes from specified package and its subpackages to some other package, and will rewrite affected bytecode.

So, during composing of "uber JAR" for Neo4j - you can configure Shade Plugin to move classes of Lucene 3 to some other package, e.g.:

org.apache.lucene.* -> shaded_3_6_2.org.apache.lucene.*

(Luckily, it seems that Neo4j doesn't use reflection, in application to Lucene stuff).

So, you can create empty maven project with following pom.xml:

    <project
    xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>my.hack</groupId>
    <artifactId>uber-neo4j</artifactId>
    <version>1.9.3</version>
    <packaging>jar</packaging>
    <name>uber-neo4j</name>

    <properties>
        <neo4j-version>1.9.3</neo4j-version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.neo4j</groupId>
            <artifactId>neo4j</artifactId>
            <version>${neo4j-version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.lucene</pattern>
                                    <shadedPattern>shaded_lucene_3_6_2.org.apache.lucene</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>neo4j-repo</id>
            <name>Neo4j Repository</name>
            <url>http://m2.neo4j.org/content/repositories/releases</url>
        </repository>
    </repositories>
</project>

Described configuration - provides an ability to generate "uber JAR" for Neo4j with renamed packages of Lucene 3 (just do mvn install).

And, finally, you can attach this stuff as a module to your maven project.

So, after this workaround - you will be able to use both: Neo4j and Lucene 4 in your project.

Just in case, here is link to GitHub repository with maven configuration for generation of "uber JAR" for Neo4j: https://github.com/lagodiuk/neo4j-uber-jar

Answer 3

There have been some changes internally from 1.8 -> 1.9. In short, a index provider must register a KernelExtensionFactory via META-INF/services, see https://github.com/neo4j/neo4j/blob/master/community/lucene-index/src/main/resources/META-INF/services/org.neo4j.kernel.extension.KernelExtensionFactory

This KernelExtensionFactory is the entry point, just checkout the Lucene 3 based implementation at https://github.com/neo4j/neo4j/tree/master/community/lucene-index.