Google DataStore with objectify - composite querie

Question

I am using Google App Engine for my REST service backend and Google dataStore as the DB and objectify for accessing datastore. One of the properties in my entity is

Map<String,String> customAttrs;

This is because I don't know before hand what all parameters can come from the client. My requirement is that I want to be able to index on these parameters in the HashMap. I looked at the following question:

GAE w/ Objectify - Can you query a HashMap?

and tried out what was mentioned in the accepted answer and it works for me, I am able to index based on the attributes in customAttrs by using

filter("customAttrs .key1", "value1")

based on whatever queryparams are passed by the client in the REST API call.

However, the problem comes in case of composite queries. I don't know much about it but from what I understand, if you combine a sorting or an inequality filter with another filter, you have to define the combination in a file called datastore-indexes.xml . So one of fields in the entity is creationTime, and when I do any query which returns me a list of users, I want to sort it by creationTime.

So let's say one the keys in the Map customAttrs is city. Now if I just want to filter all users where say city=London, it works fine for me. However, if I try to get all users where city=London sorted by creationTime, it returns an exception and does not work. It works only if I add the following in the file datastore-indexes.xml:

<datastore-index kind="User" ancestor="false" source="manual">
        <property name="customAttrs.city" direction="asc" />
        <property name="creationTime" direction="asc" />
    </datastore-index>

Now this would defeat the purpose of using HashMap since here I need to know the fields inside customAttrs before hand.

So is there any way of performing composite queries without having to know the fields before hand?

EDIT:

As mentioned in the answer by Sai, that there is no way to perform these type of queries, without adding it before-hand and adding it in datastrore-indexes.xml.

I thought of another approach, but I am facing different issue there.

In this approach, instead of defining the property as

Map<String,String> customAttrs;

I have defined it as a list of strings

List<String> customAttrs;

Here the strings which are stored are of the form "key:value". For example, the list for an entity might be "city:abc","country:xyz","name:123"

In this case, I just need to define the following in my datastore-indexes.xml:

<datastore-index kind="User" ancestor="false" source="manual">
        <property name="customAttrs" direction="asc" />
        <property name="creationTime" direction="asc" />
    </datastore-index>

which is fine for me, since I am nowhere hard-coding/pre-defining what the attributes, could be. I am able to filter by city=abc by using

filter("customAttrs","city:abc")

With this approach I can filter by any one of the parameters as well as sort the results by creationTime.

But the problem with this approach is that I am able to do only one type of custom filter. Meaning if I want to filter where city is abc and country is xyz( this works if I use the Map approach), I am not able to do it. Is there any way to do it. So basically my query is that if an entity has a property which is List, is there a way to filter all those entities which have the strings "str1" AND "str2" in the List?

Answer 1

For your latest question on filtering over Array properties -

It is possible to do an AND query on an Array type to see if it contains two or more elements. The GQL would like -

SELECT * FROM MyEntity WHERE ArrayProperty='value1' AND ArrayProperty='value2'

Answer 2

This has nothing to do with Objectfy, but the Datastore itself. Any time your query searches on multiple properties or has sorting like you mentioned (search and sort on two different properties), the Datastore expects a matching composite index. Datastore is designed so that all queries get their results from a single index for performance. That said, if you allow dynamic properties whose names are not aware ahead of time, there is really no way to perform the queries like you are wanting to do. If it is an option, your application should limit the querying/sorting on these dynamic properties so they do not require a composite index. Otherwise, you will have to explore alternative options such as full text search (either AppEngine's Full Text Search or Elastic Search) and see if they would help with your use case.