Optimizing Lucene performance

Question

What are the various ways of optimizing Lucene performance?

Shall I use caching API to store my lucene search query so that I save on the overhead of building the query again?

Answer 1

I have found that the best answer to a performance question is to profile it. Guidelines are great, but there is so many variables that can impact performance such as the size of your dataset, the types of queries you are doing, datatypes, etc.

Get the Netbeans profiler or something similar and try it out different ways. Use the articles linked to by Mitch, but make sure you actually test what helps and what (often surprisingly) hurts.

There is also a good chance that any performance differences you can get from Lucene will be minor compared to performance improvements in your code. The profiler will point that out as well.

Answer 2

Have you looked at

Lucene Optimization Tip: Reuse Searcher

Advanced Text Indexing with Lucene

Should an index be optimised after incremental indexes in Lucene?

Answer 3

Cheat. Use RAMDirectory to load the entire index into the ram. Afterwards, everything is blazing fast. :)

Answer 4

Quick tips:

• Keep the size of the index small. Eliminate norms, Term vectors when not needed. Set Store flag for a field only if it a must.
• Obvious, but oft-repeated mistake. Create only one instance of Searcher and reuse.
• Keep in the index on fast disks. RAM, if you are paranoid.

Answer 5

Lots of dead links in here.

These (somewhat official) resources are where I would start:

http://wiki.apache.org/lucene-java/ImproveIndexingSpeed

http://wiki.apache.org/lucene-java/ImproveSearchingSpeed

Answer 6

For 64 bit machines use MMapDirectory instead of RAMDirectory as very well explained here by one of the core Lucene committers.