Refreshing Caches while under load with Spring/EHC

Question

I have a caching issue on a Spring multi-threaded web service with a database backend and EHCache-based caching. The service has many clients all requesting the same object again and again, with dozens of requests per seconds. There is only a couple of objects that are requested that frequently, with a large number of other objects being requested infrequently. The objects can change every couple of minutes, so the cache's TTL is set to a minute. Loading an object from the database is slow and takes at least several seconds.

At first I used a naive implementation to get the object:

1. Check whether the object is in the cache.
2. If yes, return it from the cache.
3. If not, load it from the database, put it in the cache and return it.

This was working well when initially testing it locally. But performance testing on a faster server showed some pretty bad load spikes every time one of the more frequently requested objects expires in the cache. When this happens, for the next 10 seconds all requests for that object would result in database loads, until the first thread finished the database load and put the new object into the cache. The result was a short but very high load on the database, and a lot of users who need to wait for the request to finish.

My current implementation improves the database load by tracking whether which object are currently being loaded:

1. Check whether the object is cached.
2. If yes, return it from the cache.
3. If not, check whether the object is currently being loaded.
4. If yes, wait for the other thread's load to complete, get the new object from the cache and return it.
5. If no, put the object into the list of loading objects, put it into the cache when finished and return it.

With this implementation, even when the object expires, there is only one database operation. And, because of the lower database load, it will also finish sooner. But it still means that all users who request the object during the object load need to wait.

What I would really want is that only the first thread waits for the database load, and all others just return the 'expired' object while the object is being loaded. Response time is more important for me than the fact that the object is a few seconds too old.

Alternatively I could refresh the cache asynchronously when I notice that an object will expire in a few seconds. That's closer to EHCache's single TTL model and would mean that no one needs to wait for the database load

My real question is: before I re-invent the wheel, is there any existing framework that already implements something like this (in a Spring/EHCache environment)? Or maybe support for this already exists somewhere in Spring/EHCache and I just can't find the right option?

Answer 1

There are two Ehcache provided constructs that could help you:

1. Refresh ahead
2. Scheduled refresh

Both require you to change the way you interact with your cache as they require a CacheLoader to be configured.

Unfortunately, I can't find online documentation that shows example for the second option. It allows to refresh cache entries using Quartz to schedule it. It can also refresh only a subset of the keys, based on a key generator. Have a look at classes in package net.sf.ehcache.constructs.scheduledrefresh

Answer 2

Your design is flawed since the second thread can't get any "expired" object from the cache since there is none (as per step #2: Return immediately, when the object is in the cache).

Workarounds:

1. 10 seconds to load a single object is way too long. Check your SQL and try to optimize it.

2. Cache objects longer and run update threads which query for new states of objects in the database. That means thread #1 just triggers some background work which eventually refreshes the object in the cache. Drawback: The cache must be big enough to keep most of the objects in memory at all times. Otherwise the "load object for the first time" will be too visible.

3. Display the web page without loading the objects and load them with AJAX requests in the background. Update the web page as objects become available. Depending on how useful your site is when not everything is ready at once, this might be good balance between responsiveness and accuracy.

4. Improve loading of objects. Create "view" tables which contain all the data necessary to display a single object in each row. Update these rows when you make changes to the "real" (normalized) objects. The "view cache" is populated from this table only. That makes loading objects very fast at the expense of changes to the data model. See "Command-query separation" for an extreme solution.

5. Try to denormalize your data model a bit to reduce the number of joins necessary to load a single object. Alternatively, cache some objects which you would normally join and do the filtering/aggregation on the web server.

6. When updating an object, trigger a refresh of the cache. Chances are that someone will want to see this object, soon. This approach works best when people manually edit the objects and least, when changes are randomly triggered by outside systems (news tickers, stock quotes, etc).

7. If you only need a lot of joins to display all the details, try to load the overview and then use a second cache for details which you can then load in a second thread. Together with AJAX, you can display an overview of the object quickly which will buy you some goodwill to wait for the details.