Azure Event Hubs Changing Partition Key

Question

An application sets the value of EventData.PartitionKey to a new Guid upon start-up. For each new deployment, the Partition Key will therefore change.

I understand that Event Hubs leverage a hashing mechanism in order to route messages to specific Partitions. Does regenerating the Partition Key impede, or affect, this mechanism in any detrimental way?

I notice from time to time that messages do not appear in the Event Hub (regardless of how much time has passed) after multiple deployments, despite the fact that the underlying EventHubClient.SendBatchAsync method does not throw an Exception. The behaviour appears to correct itself arbitrarily.

Answer 1

The impact would be: each time your App restarts - the messages could land in an entirely different Partition - because, the new Guid could be hashed to a different EventHub Partition.

This wouldn't hurt EventHub performance in any way. You can generate as many PartitionKey's as you need.

But, the Application consuming these events would be impacted: typically there would be one Worker processing the events out of a Single-Partition (EventHubs partitions are the Unit-of-Scale for the event-processers processing events out of EventHubs). When your App started with Guid1 - it might be hashed to Partition1 which is being processed by ProcessorInstance1, but, when the App restarts - it would generate Guid2 which might be hashed to Partition10 which is being processed by ProcessorInstance10. The principle behind any Application using PartitionKey is Correlation - all EventData's using the same PartitionKey would land on Same EventHub partition. But, this key is being reset - which defeats the whole purpose.

All SendBatch Operations will throw if the operation doesn't succeed. This is the fundamental guarantee & an impossible SLA breach - if it doesn't throw. I believe, from the symptoms you explained, after an App deployment, since the new Guid will result into events landing on different partitions - you might be receiving events out of a specific partition & could be missing to see them & it could auto-correct if you generate another Guid which maps to old Partition... I would strongly recommend to fix the Map of partitionKey to a randomGuid.

more on Event Hubs...