Web Service Architecture: Redis (as cache) & Postg

问题:

I'm developing a Java REST API that uses client data from a postgreSQL database.

The numbers: . About 600 clients at the beginning . Some of them doing requests every few seconds

Because clients pay per request, we need to control if their number of successful requests reach their limit, and as querying postgresql data (update the value of 'hitsCounter' field) after every request is bad in terms of performance, we are thinking about implementing a cache system with redis.

The idea: After a client does his first request, we retrieve his data from postgresql and store it into redis cache. Then work with this cache-data, for example incrementing the 'hitsCounter' key value, till the client stops doing requests. In parallel, every few minutes a background process persist data from redis cache to db tables, so at the end we have the updated data back to postgresql, and we can deal with them in the future.

I think it obviously increase performance, but I'm not sure about this "background process". An option is to check the TTL of the cache elements and if it's minor than some value (it means client has finished doing requests), persist the data.

I would love to hear some opinions about this. Is this a good idea? Do you know some better alternatives?

回答1:

Perfectly reasonable idea, but you've not mentioned any measurements you've made. What is the bottleneck in your target hardware with your target transaction levels? Without knowing that, you can't say.

You could use an unlogged table perhaps. Just insert a row with every query, then summarise every 5 minutes, clearing out old data. Then again, with HOT updates, and say 75% fill-factor maybe updates are more efficient. I don't know (and nor do you) we haven't measured it.

Not enough? Stick it on its own tablespace on ssd.

Not enough? Stick it on its own vm/machine.

Not enough? Just write the damn stuff to flat files on each front-end box and batch the data once a minute into the database.

Also - how much are they paying per query? Do you care if power fails and you lose five seconds of query logs? Do you need to be able to reproduce receipts for each query with originating details and a timestamp?

回答2:

I think your approach is a good idea.

Personally, I'd make a new Thread that runs forever (while (true)). I wouldn't make two threads thou: The same thread that counts the requests stores the data into the db.

--Thread

while (true)
 { 
   if (newHit)
   {
      hits++;
      //code..
      if (hits==30) //for example
        storeDatainDB(dataInCache); 
   }   
 }