How to consume events delivered by Azure Event Gri

问题:

Basically what I understood from few Azure topics is as below:
1) Azure Event Hub - where data is received initially and converted into events
2) Service Bus- acting as a queue
3) Azure Event Grid - where events converted in hub are transferred here.

so the connection is like below:

Hub -> Service Bus -> Event Grid -> Pub Sub -> Storage

I understood this concept. My problem is I want data to be pushed from event grid to GCP (subscription / topics). So how can I establish this using PUSH method??
What do I need to develop exactly. How can I push things from grid to pubsub/subscriptions. I found this link where data is getting published into Event Grid but I want to push data from the event grid to gcp. Can anybody explain me where am I going wrong or what exactly should I start with. I am new to this and its very confusing so I just need little bit of guidance over here.

I have below doubts:
1) Is there any direct subscriber option available with event grid listener? I mean can I directly link my google storage account with this listener so, whenever there is an event triggered it will be directly pushed to my GCP account(I don't have Azure account with me right now since access issue is in progress so I can't see it that's why I am asking here)
2) Suppose I have 20 columns in my data but I want only 16 columns to be pushed in GCP so is there any customization possible while sending data from event grid/event hub to pub/sub
3) If I write custom connectors code as per the links provided in the below answers then how can I run it?? I mean where I can deploy those scripts on the cloud so that it will be triggered automatically whenever an event is triggered.
4) Can I implement webhooks in this scenario?? (as an alternative to connectors). If yes then how can I do it and on which side I need to create it?
5) Also, I read some articles and I came to know from a few guys that they experienced data loss in this entire process. So, what's the possibility over here and how can it be avoided

回答1:

Can anybody explain me where am I going wrong or what exactly should I start with.

It's right here:

so the connection is like below:
Hub -> Service Bus -> Event Grid -> Pub Sub -> Storage

Although this might be the case, it sounds very much as if you're looking at one (very) specific scenario where data flows in this exact way.

Azure Event Hub, Azure Service Bus and Azure Event Grid can work together, but can also be used completely separate from each other.

Event Grid
The purpose of Event Grid is to enable Reactive programming. Use this when you want to react to (status) changes.

Event Hubs
Event Hubs facilitate a big data pipeline. Use this when you need telemetry and distributed data streaming.

Service Bus
The purpose of Service bus is to enable High-value enterprise messaging. Use this when you want to do something like Order processing and financial transactions.

In some cases, you use the services side by side to fulfill distinct roles. For example, an ecommerce site can use Service Bus to process the order, Event Hubs to capture site telemetry, and Event Grid to respond to events like an item was shipped.

In other cases, you link them together to form an event and data pipeline. You use Event Grid to respond to events in the other services. For an example of using Event Grid with Event Hubs to migrate data to a data warehouse, see Stream big data into a data warehouse.

Taken from the very interesting and important documentation article Choose between Azure messaging services - Event Grid, Event Hubs, and Service Bus

EDIT

My problem is I want data to be pushed from event grid to GCP (subscription / topics). So how can I establish this using PUSH method??

Possibly the simplest solution is to have an Event Grid Event trigger a webhook (which might run an Azure Function or a Google Cloud Function) which in turn puts the event/message on the GCP Topic.

Publishing messages is quite well documented. There are examples on how to do so with a REST call, command-line, C#, Go, JAVA, NodeJS, PHP, Python and Ruby.

EDIT 2
What you need to do is create an Event Grid Subscription to listen to and handle Event Grid Events.
Here's an example screenshot on how to listen for events for a specific Storage Account and call a WebHook whenever such an event occurs:

Pay attention to the "Endpoint Details": that's where you can specify to, for instance, call a webhook every time an event is triggered.

回答2:

The easiest way to transfer the EventHub generated events would probably be to create an EventHub event receiver in Node.js (which you mentioned in your comments) as described here, which receives events and publishes them to Cloud Pub/Sub directly, as described in the Cloud Pub/Sub publisher documentation for Node.js.