redis key scheme for analytics

2019-05-30 01:09发布

问题:

I want to create analytics using redis - basic counters per object, per hour/day/week/month/year and total

what redis data structure would be effective for this and how can I avoid doing many calls to redis?

would it better to have each model have this sets of keys:

hash - model:<id>:years => every year has a counter

hash - model:<id>:<year> => every month has a counter

hash - model:<id>:<year>:<month> => every day has a counter

hash - model:<id>:<year>:<month>:<day> => every hour has a counter

if this scheme is correct, how would I chart this data without doing many calls to redis? I would have to loop on all year in model:<id>:years and fetch the month, then loop on the month, etc? Or I just grab all fields and their values from all keys as a batch request and then process that in the server?

回答1:

It's better to use a zset for this instead of an hash. Using timestamp as score you will be able to retrieve data for specific time range

For a date range you will use model:<id>:<year>:<month>, for an hour range (using model:<id>:<year>:<month>:<day>) and so on...

Indeed, if the date range is larger than a month (e.g. from January 1st 2014 to March 20th 2014), you will have to retrieve multiple zset (model:<id>:2014:01, model:<id>:2014:02 and model:<id>:2014:03) and merge the results.

If you really want to do a date range inside a single request, you can always store day precision data inside model:<id>:<year>. And if you want to handle date range over multiple years, you will just need to have a single zset e.g. model:<id>:byDay.

However, please note that storing historical data will increase memory consumption over time so you should already think about data retention. With Redis you can either use EXPIRE on zset or do it yourself with crons.



标签: hash redis key