Add new field or change the structure on all Fires

Question

Consider a collection of users. Each document in the collection has name and email as fields.

{
  "users": {
    "uid1": {
      "name": "Alex Saveau",
      "email": "[e-mail]"
    },
    "uid2": { ... },
    "uid3": { ... }
  }
}

Consider now that with this working Cloud Firestore database structure I launch my first version of a mobile application. Then, at some point I realize I want to include another field such as last_login.

In the code, reading all the users documents from the Firestore DB using Java would be done as

FirebaseFirestore.getInstance().collection("users").get()
        .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
            @Override
            public void onComplete(@NonNull Task<QuerySnapshot> task) {
                if (task.isSuccessful()) {
                    for (DocumentSnapshot document : task.getResult()) {
                        mUsers.add(document.toObject(User.class));
                    }
                }
            }
        });

where the class User contains now name, email and last_login.

Since the new User field (last_login) is not included in the old users stored in the DB, the application is crashing because the new User class is expecting a last_login field which is returned as null by the get() method.

What would be the best practice to include last_login in all the existing User documents of the DB without losing their data on a new version of the app? Should I run an snippet just once to do this task or are there any better approaches to the problem?

Answer 1

I wrote some routines to help automate this process back when I posted the question. I did not post them since these are a bit rudimentary and I was hoping for an elegant Firestore-based solution. Because such solution is not still available, here are the functions I wrote.

In short, we have functions for renaming a field, adding a field, or deleting a field. To rename a field, different functions are used depending on the data type. Maybe someone could generalise this better?. The function you will find below are:

• add_field: Adds a field in all documents of a collection
• delete_field: Deletes a field in all documents of a collection.
• rename_*_field: Renames a field containing a certain data type (*) in all documents of a collection. Here I include examples for String, Integer, and Date.

Add field:

public void add_field (final String key, final Object value, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {
                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> new_map = new HashMap<>();
                            new_map.put(key, value);
                            batch.update(docRef, new_map);
                        }
                        batch.commit();
                    } else {
                        // ... "Error adding field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting documents -> " + e
                }
            });
}

Delete field:

public void delete_field (final String key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> delete_field = new HashMap<>();
                            delete_field.put(key, FieldValue.delete());
                            batch.update(docRef, delete_field);
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

Rename field:

public void rename_string_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            String old_value = document.getString(old_key);

                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices ->" + e
                }
            });
}

public void rename_integer_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            int old_value = document.getDouble(old_key).intValue();
                            Integer ov = old_value;
                            if (ov != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

public void rename_date_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Date old_value = document.getDate(old_key);
                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

Answer 2

You fell into a gap of NOSQL databases: Document oriented databases do not guarantee structural integrity of the data (as RDBMS do)

The deal is:

• in an RDBMS all stored data have the same structure at any given time (within the same instance or cluster). When changing the structure (ER-diagram) you have to migrate the data for all existing records which costs time and efford.

  In result you application can be optimised for the current version of the data structure.

• in a Document oriented database each record is an independend "Page" with its own independend structure. If you change the struckture it only applies to new documents. So you don't need to migrate the exiting data.

  In result you application must be able to deal with all versions of your data structure you ever used in your current database.

I don't know about firebase in detail but in general you never update a document in a NOSQL database. You only create a new version of the document. So even if you update all documents your application must be prepaed to deal with the "old" data structure...

Answer 3

To solve this, you need to update each user to have the new property and for that I recommend you to use a Map. If you are using a model class when you are creating the users as explained in my answer from this post, to update all users, just iterate over the users collection amd use the following code:

Map<String, Object> map = new HashMap<>();
map.put("timestamp", FieldValue.serverTimestamp());
userDocumentReference.set(map, SetOptions.merge());

Answer 4

I am guessing that last_login is a primitive data type, maybe a long to hold a timestamp. An auto-generated setter would look like this:

private long last_login;

public void setLast_login(long last_login) {
    this.last_login = last_login;
}

This leads to a crash when old documents that lack the field are fetched due to null assignment to a variable of a primitive data type. One way around it is to modify your setter to pass in a variable of the equivalent wrapper class - Long instead of long in this case, and put a null check in the setter.

private long last_login;

public void setLast_login(Long last_login) {
    if(last_login != null) {
        this.last_login = last_login;
    }
}

The cost of avoiding the null pointer exception is the boxing-unboxing overhead.