How to set a rest api endpoint as Storm spout?

问题:

I would like to setup a rest api endpoint as a storm spout i.e. whenever a request is made to the endpoint url (for ex. http://example.com/search?q=test) then I would like to execute a storm topology with certain bolts.

The problem is that not all the time requests are made also, the spout's nextTuple() method should be called at a certain interval of time and there is no idea whatsoever about this interval. And how could I also listen to a http url for request and execute the topology whenever a request is made? How to control concurrent requests and how could I determine the no. of workers to execute the topology because traffic to that url is uncertain.

One alternative workaround, is I thought of pushing the requests to the kafka queue as and when a request is made and then using the kafka spout. Any other idea on doing this without using kafka?

I need to do two jobs: process the request into json format and then produce the json string to kafka queue. Is storm a better solution for this?

Do I have to consider an alternative framework like Spark to do this?

回答1:

You need to have a look at Storm DRPC. In easy words, Storm DRPC is like a function, you provide params and you will get a result. Your Storm topology will be utilised to provide the answer.

Let your REST endpoint send the request to Storm DRCP.