Full Text Search API for App Engine (Java)

Question

I have been trying to use the experimental Search API for the Google AppEngine. I am using SDK 1.6.6. So far, I could add entries to an index and search it in my development environment. When I deploy my application, I can still add entries to the index, but when I try to perform a search, I get the following exception log:

br.com.qmagico.server.service.search.GenericSearchService search: Busca com query 'soma' falhou
com.google.appengine.api.search.SearchException: 
at com.google.appengine.api.search.IndexImpl$3.convertException(IndexImpl.java:268)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106)
at com.google.appengine.api.search.FutureHelper.getInternal(FutureHelper.java:74)
at com.google.appengine.api.search.FutureHelper.quietGet(FutureHelper.java:33)
at com.google.appengine.api.search.IndexImpl.search(IndexImpl.java:390)
at br.com.qmagico.server.service.search.GenericSearchService.search(GenericSearchService.java:109)
at br.com.qmagico.server.service.search.AulaSearchService.search(AulaSearchService.java:50)
at br.com.qmagico.server.handler.GetAulasHandler.executeLoggedException(GetAulasHandler.java:45)
at br.com.qmagico.server.handler.GetAulasHandler.executeLoggedException(GetAulasHandler.java:1)
at br.com.qmagico.server.QmActionHandler.execute(QmActionHandler.java:23)
at com.gwtplatform.dispatch.server.AbstractDispatchImpl.doExecute(AbstractDispatchImpl.java:153)
at com.gwtplatform.dispatch.server.AbstractDispatchImpl.execute(AbstractDispatchImpl.java:111)
at com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl.execute(AbstractDispatchServiceImpl.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_$1.run(Method_.java:165)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_.privilegedInvoke(Method_.java:163)
at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_.invoke_(Method_.java:124)
at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_.invoke(Method_.java:43)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:102)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:477)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)

The piece of code performing this search is this one:

public List<Long> search(Index index, String queryStr, String sortField) {
    ArrayList<Long> ids = new ArrayList<Long>();
    if (queryStr != null && !queryStr.isEmpty()) {
        int limit = 5;
        try {
            QueryOptions.Builder builder = QueryOptions.newBuilder().setLimit(limit);
            if (sortField != null) {
                builder = builder.setSortOptions(SortOptions.newBuilder()
                    .addSortExpression(SortExpression.newBuilder()
                        .setExpression(sortField)
                        .setDefaultValue("")
                        .setDirection(SortExpression.SortDirection.DESCENDING))
                    .build());
            }
            QueryOptions queryOptions = builder.build();

            Query query = Query.newBuilder()
                .setOptions(queryOptions)
                .build(queryStr);
            LOG.info("Enviando query " + query);
            Results<ScoredDocument> results = index.search(query);
            for (ScoredDocument scoredDoc : results) {
                ids.add(scoredDoc.getOnlyField("id").getNumber().longValue());
            }
        } catch (RuntimeException e) {
            LOG.log(Level.SEVERE, "Busca com query '" + queryStr + "' falhou", e);
        }
    }
    return ids;
}

Does anyone one what I should do to fix this?

Thx!

Answer 1

If you think sorting is your issue then you may want to look at presorting your documents in the index using Document rank. The rank field on documents is by default set to the number of seconds since Jan 1 2011. However, you can supply a integer rank value. For example, if you wanted to sort a product catalog on price (low to high, and high to low). The rank is used to presort the documents in the index.

public Index getIndex(String indexName) {
    IndexSpec indexSpec = IndexSpec.newBuilder().setName(indexName).build();
    return SearchServiceFactory.getSearchService().getIndex(indexSpec);
}

Document doc = Document.newBuilder()
    .addField(Field.newBuilder().setName('name').setText("some text"))
    .setRank(functionToGetPrice())
    .build();

getIndex("productHighToLow").put(doc);

doc = Document.newBuilder()
    .addField(Field.newBuilder().setName('name').setText("some text"))
    .setRank(MAXINT-functionToGetPrice())
    .build();

getIndex("productLowToHigh").put(doc);

Then when you search your index, if the user has selected "price low to high" and queried for "LED TV", then you can simply issue a query like

getIndex("productLowToHigh").search("LED TV");

and it will be faster and semantically correct even for large (product catalog) indexes.

Answer 2

Did you tried removing SortOptions ? I got same error when I set SortOptions. I think this is a bug.