How do you properly add/manipulate thousands of ch

Question

This further to my previous question on handling large numbers of objects in BigTables/JDO.

Assuming a TransactionAccount could end up with as many as 10,000 objects in its transactions list, how does this work with Goodle app engine?

How do you add objects to such a large list without the whole list being loaded into memory? (The assumption is that 10,000 objects shouldn't be loaded into memory?)

I am not trying to ask you how to do my homework, I just have no idea where to start to solve this, the app engine documentation and google searching is not helping :(

// example only, not meant to compile
@PersistenceCapable
public class TransactionAccount {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    public Key key;
    private long balance;
    private long transactionCount;
    @Element(dependent = "true")
    private List<Transaction> transactions = new ArrayList<Transaction>();
    ....
    public long getBalance() { return balance; }
}

@PersistenceCapable
private class Transaction {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    public Key key;
    public Date date;
    public long amount;
}

This question is raised but not resolved in the following google groups post.

Answer 1

try marking the transactions property @NotPersistent, so that it's not stored in the datastore at all. you can get the Transaction entities for a given TransactionAccount with an ancestor query (more in this thread). with that, you should be able to store arbitrarily many transactions for a given account, since they're not all stored in the account entity.

a less drastic measure would be to mark the transactions property unindexed with this annotation:

@Extension(vendorName = "datanucleus", key = "gae.unindexed", value="true") 

the account's transactions would still be stored in the list, but they wouldn't be indexed, which would make it a bit more feasible. still, you'd hit the 1MB entity size limit around 10-100k transactions, which wouldn't be a problem if you use @NotPersistent.