Returning entities in Rest API with Spring

Question

Creating a restful api for a web application in Spring is pretty easy. Let's say we have a Movie entity, with a name, year, list of genres and list of actors. In order to return a list of all movies in json format, we just create a method in some controller that will query a database and return a list as a body of ResponseEntity. Spring will magically serialize it, and all works great :)

But, what if I, in some case, want that list of actors in a movie to be serialized, and not in other? And in some other case, alongside the fields of the movie class, I need to add some other properties, for each movie in the list, which values are dynamically generated?

My current solution is to use @JsonIgnore on some fields or to create a MovieResponse class with fields like in Movie class and additional fields that are needed, and to convert from Movie to MovieResponse class each time.

Is there a better way to do this?

Answer 1

The point of the JSONIgnore annotation is to tell the DispatcherServlet (or whatever component in Spring handles rendering the response) to ignore certain fields if those fields are null or otherwise omitted.

This can provide you with some flexibility in terms of what data you expose to the client in certain cases.

Downside to JSONIgnore:

However, there are some downsides to using this annotation that I've recently encountered in my own projects. This applies mainly to the PUT method and cases where the object that your controller serializes data to is the same object that is used to store that data in the database.

The PUT method implies that you're either creating a new collection on the server or are replacing a collection on the server with the new collection you're updating.

Example of Replacing a Collection on the server:

Imagine that you're making a PUT request to your server, and the RequestBody contains a serialized Movie entity, but this Movie entity contains no actors because you've omitted them! Later on down the road, you implement a new feature that allows your users to edit and correct spelling errors in the Movie description, and you use PUT to send the Movie entity back to the server, and you update the database.

But, let's say that -- because it's been so long since you added JSONIgnore to your objects -- you've forgotten that certain fields are optional. In the client side, you forget to include the collection of actors, and now your user accidentally overwrites Movie A with actors B, C, and D, with Movie A with no actors whatsoever!

Why is JSONIgnore opt-in?

It stands to reason that the intention behind forcing you to opt-out of making certain fields required is precisely so that these types of data integrity issues are avoided. In a world where you're not using JSONIgnore, you guarantee that your data can never be replaced with partial data unless you explicitly set that data yourself. With JSONIgnore, you remove these safeguards.

With that said, JSONIgnore is very valuable, and I use it myself in precisely the same manner to reduce the size of the payload sent to the client. However, I'm beginning to rethink this strategy and instead opt for one where I use POJO classes in a separate layer for sending data to the frontend than what I use to interact with the database.

Possible Better Setup?:

The ideal setup -- from my experience dealing with this particular problem -- is to use Constructor injection for your Entity objects instead of setters. Force yourself to have to pass in every parameter at instantiation time so that your entities are never partially filled. If you try to partially fill them, the compiler stops you from doing something you may regret.

For sending data to the client side, where you may want to omit certain pieces of data, you could use a separate, disconnected entity POJO, or use a JSONObject from org.json.

When sending data from the client to the server, your frontend entity objects receive the data from the model database layer, partially or full, since you don't really care if the frontend gets partial data. But then when storing the data in the datastore, you would first fetch the already-stored object from the datastore, update its properties, and then store it back in the datastore. In other words, if you were missing the actors, it wouldn't matter because the object you're updating from the datastore already has the actors assigned to it's properties. Thus, you only replace the fields that you explicitly intend to replace.

While there would be more maintenance overhead and complexity to this setup, you would gain a powerful advantage: The Java compiler would have your back! It won't let you or even a hapless colleague do anything in the code that might compromise the data in the datastore. If you attempt to create an entity on the fly in your model layer, you'll be forced to use the constructor, and forced to provide all of the data. If you don't have all of the data and cannot instantiate the object, then you'll either need to pass empty values (which should signal a red flag to you) or fetch that data from the datastore first.

Answer 2

I ran into this problem, and really wanted to keep using @JsonIgnore, but also use the entities/POJO's to use in the JSON calls.

After a lot of digging I came up with the solution of automatically retrieving the ignored fields from the database, on every call of the object mapper.

Ofcourse there are some requirements which are needed for this solution. Like you have to use the repository, but in my case this works just the way I need it.

For this to work you need to make sure the ObjectMapper in MappingJackson2HttpMessageConverter is intercepted and the fields marked with @JsonIgnore are filled. Therefore we need our own MappingJackson2HttpMessageConverter bean:

public class MvcConfig extends WebMvcConfigurerAdapter {
    @Override
    public void extendMessageConverters(List<HttpMessageConverter<?>> converters) {
        for (HttpMessageConverter converter : converters) {
            if (converter instanceof MappingJackson2HttpMessageConverter) {
                ((MappingJackson2HttpMessageConverter)converter).setObjectMapper(objectMapper());
            }
        }
    }

    @Bean
    public ObjectMapper objectMapper() {
        ObjectMapper objectMapper = new FillIgnoredFieldsObjectMapper();
        Jackson2ObjectMapperBuilder.json().configure(objectMapper);

        return objectMapper;
    }
}

Each JSON request is than converted into an object by our own objectMapper, which fills the ignored fields by retrieving them from the repository:

    /**
     * Created by Sander Agricola on 18-3-2015.
     *
     * When fields or setters are marked as @JsonIgnore, the field is not read from the JSON and thus left empty in the object
     * When the object is a persisted entity it might get stored without these fields and overwriting the properties
     * which where set in previous calls.
     *
     * To overcome this property entities with ignored fields are detected. The same object is than retrieved from the
     * repository and all ignored fields are copied from the database object to the new object.
     */
    @Component
    public class FillIgnoredFieldsObjectMapper extends ObjectMapper {
        final static Logger logger = LoggerFactory.getLogger(FillIgnoredFieldsObjectMapper.class);

        @Autowired
        ListableBeanFactory listableBeanFactory;

        @Override
        protected Object _readValue(DeserializationConfig cfg, JsonParser jp, JavaType valueType) throws IOException, JsonParseException, JsonMappingException {
            Object result = super._readValue(cfg, jp, valueType);
            fillIgnoredFields(result);

            return result;
        }

        @Override
        protected Object _readMapAndClose(JsonParser jp, JavaType valueType) throws IOException, JsonParseException, JsonMappingException {
            Object result = super._readMapAndClose(jp, valueType);
            fillIgnoredFields(result);

            return result;
        }

        /**
         * Find all ignored fields in the object, and fill them with the value as it is in the database
         * @param resultObject Object as it was deserialized from the JSON values
         */
        public void fillIgnoredFields(Object resultObject) {
            Class c = resultObject.getClass();
            if (!objectIsPersistedEntity(c)) {
                return;
            }

            List ignoredFields = findIgnoredFields(c);
            if (ignoredFields.isEmpty()) {
                return;
            }

            Field idField = findIdField(c);
            if (idField == null || getValue(resultObject, idField) == null) {
                return;
            }

            CrudRepository repository = findRepositoryForClass(c);
            if (repository == null) {
                return;
            }

            //All lights are green: fill the ignored fields with the persisted values
            fillIgnoredFields(resultObject, ignoredFields, idField, repository);
        }

        /**
         * Fill the ignored fields with the persisted values
         *
         * @param object Object as it was deserialized from the JSON values
         * @param ignoredFields List with fields which are marked as JsonIgnore
         * @param idField The id field of the entity
         * @param repository The repository for the entity
         */
        private void fillIgnoredFields(Object object, List ignoredFields, Field idField, CrudRepository repository) {
            logger.debug("Object {} contains fields with @JsonIgnore annotations, retrieving their value from database", object.getClass().getName());

            try {
                Object storedObject = getStoredObject(getValue(object, idField), repository);
                if (storedObject == null) {
                    return;
                }

                for (Field field : ignoredFields) {
                    field.set(object, getValue(storedObject, field));
                }
            } catch (IllegalAccessException e) {
                logger.error("Unable to fill ignored fields", e);
            }
        }

        /**
         * Get the persisted object from database.
         *
         * @param id The id of the object (most of the time an int or string)
         * @param repository The The repository for the entity
         * @return The object as it is in the database
         * @throws IllegalAccessException
         */
        @SuppressWarnings("unchecked")
        private Object getStoredObject(Object id, CrudRepository repository) throws IllegalAccessException {
            return repository.findOne((Serializable)id);
        }

        /**
         * Get the value of a field for an object
         *
         * @param object Object with values
         * @param field The field we want to retrieve
         * @return The value of the field in the object
         */
        private Object getValue(Object object, Field field) {
            try {
                field.setAccessible(true);
                return field.get(object);
            } catch (IllegalAccessException e) {
                logger.error("Unable to access field value", e);
                return null;
            }
        }

        /**
         * Test if the object is a persisted entity
         * @param c The class of the object
         * @return true when it has an @Entity annotation
         */
        private boolean objectIsPersistedEntity(Class c) {
            return c.isAnnotationPresent(Entity.class);
        }

        /**
         * Find the right repository for the class. Needed to retrieve the persisted object from database
         *
         * @param c The class of the object
         * @return The (Crud)repository for the class.
         */
        private CrudRepository findRepositoryForClass(Class c) {
            return (CrudRepository)new Repositories(listableBeanFactory).getRepositoryFor(c);
        }

        /**
         * Find the Id field of the object, the Id field is the field with the @Id annotation
         *
         * @param c The class of the object
         * @return the id field
         */
        private Field findIdField(Class c) {
            for (Field field : c.getDeclaredFields()) {
                if (field.isAnnotationPresent(Id.class)) {
                    return field;
                }
            }

            return null;
        }

        /**
         * Find a list of all fields which are ignored by json.
         * In some cases the field itself is not ignored, but the setter is. In this case this field is also returned.
         *
         * @param c The class of the object
         * @return List with ignored fields
         */
        private List findIgnoredFields(Class c) {
            List ignoredFields = new ArrayList();
            for (Field field : c.getDeclaredFields()) {
                //Test if the field is ignored, or the setter is ignored.
                //When the field is ignored it might be overridden by the setter (by adding @JsonProperty to the setter)
                if (fieldIsIgnored(field) ? setterDoesNotOverrideIgnore(field) : setterIsIgnored(field)) {
                    ignoredFields.add(field);
                }
            }
            return ignoredFields;
        }

        /**
         * @param field The field we want to retrieve
         * @return True when the field is ignored by json
         */
        private boolean fieldIsIgnored(Field field) {
            return field.isAnnotationPresent(JsonIgnore.class);
        }

        /**
         * @param field The field we want to retrieve
         * @return true when the setter is ignored by json
         */
        private boolean setterIsIgnored(Field field) {
            return annotationPresentAtSetter(field, JsonIgnore.class);
        }

        /**
         * @param field The field we want to retrieve
         * @return true when the setter is NOT ignored by json, overriding the property of the field.
         */
        private boolean setterDoesNotOverrideIgnore(Field field) {
            return !annotationPresentAtSetter(field, JsonProperty.class);
        }

        /**
         * Test if an annotation is present at the setter of a field.
         *
         * @param field The field we want to retrieve
         * @param annotation The annotation looking for
         * @return true when the annotation is present
         */
        private boolean annotationPresentAtSetter(Field field, Class annotation) {
            try {
                Method setter = getSetterForField(field);
                return setter.isAnnotationPresent(annotation);
            } catch (NoSuchMethodException e) {
                return false;
            }
        }

        /**
         * Get the setter for the field. The setter is found based on the name with "set" in front of it.
         * The type of the field must be the only parameter for the method
         *
         * @param field The field we want to retrieve
         * @return Setter for the field
         * @throws NoSuchMethodException
         */
        @SuppressWarnings("unchecked")
        private Method getSetterForField(Field field) throws NoSuchMethodException {
            Class c = field.getDeclaringClass();
            return c.getDeclaredMethod(getSetterName(field.getName()), field.getType());
        }

        /**
         * Build the setter name for a fieldName.
         * The Setter name is the name of the field with "set" in front of it. The first character of the field
         * is set to uppercase;
         *
         * @param fieldName The name of the field
         * @return The name of the setter
         */
        private String getSetterName(String fieldName) {
            return String.format("set%C%s", fieldName.charAt(0), fieldName.substring(1));
        }
    }

Maybe not the most clean solution in all cases, but in my case it does the trick just the way I want it to work.