Making a high number of http GET requests with few

Question

Unfortunately, JDK's HttpUrlConnection or UrlConnection do not run asynchronously. So, if HTTP connections hang for any reason, you will end up loosing a thread. Plus, Selector's multiplexing allows for using only a few threads for making a high number of HTTP GET calls. So, I have been developing a code that uses NIO's Selector and makes HTTP GET. So, the core works well except that to fully support HTTP 1.1, I have to support Keep-Alive, chunked-transfer-mode, and several other things that HTTP 1.1 supports.

So, I have been also looking to do this with Netty, but I have not been able to find an example, which shows how to make several HTTP GET requests using a single thread.

So, I would appreciate it if someone could point me to a netty example or any other appropriate library. Thanks

Answer 1

Netty's Channel EventLoop model manage a minimum number of threads to support a maximum number of client connections. It is baked into the framework.

There are a few HTTP examples in the API Docs (there is a folder called example in the source files). There are also a few HTTP examples on GitHub based on the examples from the Manning book Netty in Action. Chapter 15 - EventLoop and Thread-Model of this book discusses all this in detail.