Real-time pubsub chat with history via websockets

Question

I'm interested in creating what Disqus have done with their commenting system: http://highscalability.com/blog/2014/5/7/update-on-disqus-its-still-about-realtime-but-go-demolishes.html

The most impressive part of infrastructure is Nginx push stream module:

Still runs on 5 machines Nginx machines.

Uses NginxPushStream, which supprts EventSource, WebSocket, Long Polling, and Forever Iframe.

All users are connected to these machines.

On a normal day each machine sees 3200 connections/s, 1 million connections, 150K packets/s TX and 130K packets/s RX, 150 mbits/s TX and 80 mbits/s RC, with <15ms delay end-to-end (which is faster than Javascript can render a comment)

Had many issues with resource exhaustion at first. The configuration for Nginx and the OS are given that help alleviate the problems, tuning them to handle a scenario with many connections moving little data.

Obviously this Nginx module doesn't support data storage. Only in-memory mechanism supported by push_stream_store_messages directive, but as author said:

The main target to stored messages is to deliver the message to a subscriber that was offline when the message was published.

It is clear that Disqus don't publish messages to Nginx directly but rather though the Go backend that manages to store messages in Redis followed by publishing via internal POST to Nginx which keeps subscribers.

Does anyone have an experience with fetching messages history through Redis on a page load before even we use push stream module for newer messages? Do you have to push old messages history in one go or render as plain HTML followed by pubsub for new messages that appear after page is loaded?

The logic needs to be decoupled as much as possible. I don't intend to introduce blocking mechanism between user and Nginx for real time message communication. Would that be a good solution below?

1. Client pushes the message from the web page (via websockets)
2. Ajax request goes straight to push stream location and on Ajax complete callback it requests backend to store message in Redis (in the other direction it gets locked with backend)
3. Once user refreshes the page backend fetches Redis list and display the history
4. User can see the history and can post new messages

It requires only 2 backend requests to be developed: accept message and store in Redis, fetch the data and display on page load. Preferably light non-blocking backend required, like Lua module or even HTTP interface to Redis called Webdis.

I'd like to know smart people opinion on that mechanism from architecture point of view, no code example expected.