Why did Lucene IndexWriter did not update the inde

Question

I have a strange problem with a Web Project using Lucene. I have written a bean which writes the data of a business object into Lucene. The business data is provided by the web front end. This all works very fine. But in some rarely cases the business object is not added into the index. It seems that this happens in cases when multiple users are accessing the bean.

My bean uses static methods creating the Lucene IndexWriter and creating a Lucene Document on base of my business object. I thought making all methods static is enough to become thread save. But now I wonder if it is necessary to make my update method also 'synchronized' to avoid conflicts with other threads? In the cases when the object is not written into the index there are no exceptions thrown from lucene. So I can not say what happens behind my layer.

My update method looks like this:

public static boolean updateWorklist(ItemCollection workitem) throws PluginException {

        IndexWriter awriter = null;
        // try loading imixs-search properties
        Properties prop = loadProperties();
        if (prop.isEmpty())
            return false;

        try {
            awriter = createIndexWriter(prop);

            // create term
            Term term = new Term("$uniqueid", workitem.getItemValueString("$uniqueid"));
            // test if document should be indexed or not
            if (matchConditions(prop, workitem)) {
                logger.fine("add workitem '" + workitem.getItemValueString(EntityService.UNIQUEID) + "' into index");
                awriter.updateDocument(term, createDocument(workitem));
            } else {
                logger.fine("remove workitem '" + workitem.getItemValueString(EntityService.UNIQUEID) + "' into index");
                awriter.deleteDocuments(term);
            }
        } catch (IOException luceneEx) {
            // close writer!
            logger.warning(" Lucene Exception : " + luceneEx.getMessage());

            throw new PluginException(LucenePlugin.class.getSimpleName(), INVALID_INDEX,
                    "Unable to update search index", luceneEx);

        } finally {

            if (awriter != null) {
                logger.fine(" close writer");
                try {
                    awriter.close();
                } catch (CorruptIndexException e) {
                    throw new PluginException(LucenePlugin.class.getSimpleName(), INVALID_INDEX,
                            "Unable to update search index", e);
                } catch (IOException e) {
                    throw new PluginException(LucenePlugin.class.getSimpleName(), INVALID_INDEX,
                            "Unable to update search index", e);
                }

            }
        }

        logger.fine(" update worklist successfull");
        return true;
    }

.....

    public static IndexWriter createIndexWriter(Properties prop)
            throws IOException {

        /**
         * Read configuration
         */
        // String sLuceneVersion = prop.getProperty("Version", "LUCENE_45");

        String sIndexDir = prop.getProperty("lucence.indexDir");
        String sFulltextFieldList = prop
                .getProperty("lucence.fulltextFieldList");
        String sIndexFieldListAnalyse = prop
                .getProperty("lucence.indexFieldListAnalyze");
        String sIndexFieldListNoAnalyse = prop
                .getProperty("lucence.indexFieldListNoAnalyze");

        logger.fine("IndexDir:" + sIndexDir);
        logger.fine("FulltextFieldList:" + sFulltextFieldList);
        logger.fine("IndexFieldListAnalyse:" + sIndexFieldListAnalyse);
        logger.fine("IndexFieldListNoAnalyse:" + sIndexFieldListNoAnalyse);
        // compute search field list
        StringTokenizer st = new StringTokenizer(sFulltextFieldList, ",");
        searchFieldList = new ArrayList<String>();
        while (st.hasMoreElements()) {
            String sName = st.nextToken().toLowerCase();
            // do not add internal fields
            if (!"$uniqueid".equals(sName) && !"$readaccess".equals(sName))
                searchFieldList.add(sName);
        }

        // compute Index field list (Analyze)
        st = new StringTokenizer(sIndexFieldListAnalyse, ",");
        indexFieldListAnalyse = new ArrayList<String>();
        while (st.hasMoreElements()) {
            String sName = st.nextToken().toLowerCase();
            // do not add internal fields
            if (!"$uniqueid".equals(sName) && !"$readaccess".equals(sName))
                indexFieldListAnalyse.add(sName);
        }

        // compute Index field list (Analyze)
        st = new StringTokenizer(sIndexFieldListNoAnalyse, ",");
        indexFieldListNoAnalyse = new ArrayList<String>();
        while (st.hasMoreElements()) {
            String sName = st.nextToken().toLowerCase();
            // do not add internal fields
            if (!"$uniqueid".equals(sName) && !"$readaccess".equals(sName))
                indexFieldListNoAnalyse.add(sName);
        }

        /**
         * Now create a IndexWriter Instance
         */
        Directory indexDir = createIndexDirectory(prop);

        Analyzer analyzer = new StandardAnalyzer();
        IndexWriterConfig indexWriterConfig = new IndexWriterConfig(
                Version.LATEST, analyzer);

        // set the WriteLockTimeout to wait for a write lock (in milliseconds)
        // for this instance. 10 seconds!
        indexWriterConfig.setWriteLockTimeout(10000);

        return new IndexWriter(indexDir, indexWriterConfig);
    }

....

public static boolean matchConditions(Properties prop, ItemCollection aworktiem) {

    String typePattern = prop.getProperty("lucence.matchingType");
    String processIDPattern = prop.getProperty("lucence.matchingProcessID");

    String type = aworktiem.getItemValueString("Type");
    String sPid = aworktiem.getItemValueInteger("$Processid") + "";

    // test type pattern
    if (typePattern != null && !"".equals(typePattern) && !type.matches(typePattern)) {
        logger.fine("Lucene type '" + type + "' did not match pattern '" + typePattern + "'");
        return false;
    }

    // test $processid pattern
    if (processIDPattern != null && !"".equals(processIDPattern) && !sPid.matches(processIDPattern)) {
        logger.fine("Lucene $processid '" + sPid + "' did not match pattern '" + processIDPattern + "'");

        return false;
    }
    return true;
}

.... Edit 20.Jan: I added the IndexWriter method into the code example. My property file is empty and did not provide any lucene settings to the IndexWriter

Edit 21.Jan: I added the matchConditions method

Answer 1

As explained by @maksim07, the problem in my implementation was the use of static variables and methods. So the createDocument method could observe empty searchFieldList, indexFieldListAnalyse, indexFieldListNoAnalyse collections since there was a race condition.

Finally I re-implemented the code by using the singleton pattern with a separate init() method where I initialized all the values of my member variables. Using a Singleton EJB makes the usage of a singleton pattern very easy and guaranties that all method calls are synchronized by the default behavior of the EJB container.