How efficient is Google App Engine ndb.delete_mult

Question

I'm working on something to clear my database of ~10,000 entities, and my plan is to put it in a task that deletes 200 at a time using ndb.delete_multi() and then recursively calls itself again until there are no entities left.

For now, I don't have the recursion in it yet so I could run the code a few times manually and check for errors, quota use, etc. The code is:

entities = MyModel.query_all(ndb.Key('MyModel', '*defaultMyModel')).fetch(200)
key_list = ndb.put_multi(entities)
ndb.delete_multi(key_list)

All the query_all() does is query MyModel and return everything.

I've done some testing by commenting out things and running the method, and it looks like the first two lines take up the expected amount of writes (~200).

Running the third line, ndb.delete_multi(), takes up about 8% of my 50,000 daily write allowance, so about 4000 writes--20 times as many as I think it should be doing.

I've also made sure the key_list contains only 200 keys with logging.

Any ideas on why this takes up so many writes? Am I using the method wrong? Or does it just use a ton of memory? In that case, is there any way for me to do this more efficiently?

Thanks.

Answer 1

Your code example is extremely inefficient. If you are deleting large numbers of entities than you will need to batch the below but, you should be retrieving data with a keys_only query and then deleting:

from google.appengine.ext import ndb

ndb.delete_multi(
    MyModel.query().fetch(keys_only=True)
)

In regards to the number of write operations (see Andrei's answer), ensure only the fields on your model that are required to be indexed "have an index enabled".

Answer 2

When you delete an entity, the Datastore has to remove an entity and a record from an index for each indexed property as well as for each custom index. The number of writes is not dependent on which delete method you use.