Kinesis stream pending message count

问题:

I am trying to use AWS Kinesis stream for one of our data streams. I would like to monitor pending messages on my stream for ops purposes(scale downstream according to backlog), but unable to find any API that gives (approx) pending messages in my stream.

This looks strange as messages get expired after 7 days and if the producers and consumers are isolated and can't communicate, how do you know messages are expiring. How do you handle this problem?

Thanks!

回答1:

There is no such concept as "pending" message in Kinesis. All the incoming data will be placed on a shard.

Your consumer application should be in running state all the time, to keep track of changes in your stream. The application (with the help of KCL) will continue to poll "Shard Iterator" in the background, thus you will be notified about the new data when it comes.

Roughly; you can see Kinesis as a FIFO queue and the messages will disappear in a short time if you don't pop them.

If your application will process a few messages in an hour, you should think about changing your architecture. Kinesis is probably not the correct tool for you.