Redis and Node.js and Socket.io Questions

Question

I have been just learning redis and node.js There are two questions I have for which I couldn't find any satisfying answer.

My first question is about reusing redis clients within the node.js. I have found this question and answer: How to reuse redis connection in socket.io? , but it didn't satisfy me enough.

Now, if I create the redis client within the connection event, it will be spawned for each connection. So, if I have 20k concurrent users, there will be 20k redis clients.

If I put it outside of the connection event, it will be spawned only once.

The answer is saying that he creates three clients for each function, outside of the connection event.

However, from what I know MySQL that when writing an application which spawns child processes and runs in parallel, you need to create your MySQL client within the function in which you are creating child instances. If you create it outside of it, MySQL will give an error of "MySQL server has gone away" as child processes will try to use the same connection. It should be created for each child processes separately.

So, even if you create three different redis clients for each function, if you have 30k concurrent users who send 2k messages concurrently, you should run into the same problem, right? So, every "user" should have their own redis client within the connection event. Am I right? If not, how node.js or redis handles concurrent requests, differently than MySQL? If it has its own mechanism and creates something like child processes within the redis client, why we need to create three different redis clients then? One should be enough.

I hope the question was clear.

-- UPDATE --

I have found an answer for the following question. http://howtonode.org/control-flow No need to answer but my first question is still valid.

-- UPDATE --

My second question is this. I am also not that good at JS and Node.js. So, from what I know, if you need to wait for an event, you need to encapsulate the second function within the first function. (I don't know the terminology yet). Let me give an example;

socket.on('startGame', function() {
    getUser();
    socket.get('game', function (gameErr, gameId) {
        socket.get('channel', function (channelErr, channel) {
            console.log(user);
            client.get('games:' + channel + '::' + gameId + ':owner', function (err, owner) { //games:channel.32:game.14
                if(owner === user.uid) {
   //do something
                }
            });
        }
    });
});

So, if I am learning it correctly, I need to run every function within the function if I need to wait I/O answer. Otherwise, node.js's non-blocking mechanism will allow the first function to run, in this case it will get the result in parallel, but the second function might not have the result if it takes time to get. So, if you are getting a result from redis for example, and you will use the result within the second function, you have to encapsulate it within the redis get function. Otherwise second function will run without getting the result.

So, in this case, if I need to run 7 different functions and the 8. function will need the result of all of them, do I need to write them like this, recursively? Or am I missing something.

I hope this was clear too.

Thanks a lot,

Answer 1

As for the first question: The "right answer" might make you think you are good with one Connection. In reality, whenever you are doing something that is waiting on an IO, a timer, etc, you are actually making node run the waiting method on the queue. Hence, if you use only 1 single connection, you will actually limit the performance of the thread you working on ( a single CPU) to the speed of redis - which is probably a few hundreds of callbacks per second (non-redis waiting callbacks will still go on) - while this is not poor performance, there's no reason to create this kind of limitation. It is recommended to create a few (5-10) connections to avoid this issue in it's entire. This number goes up for slower databases, e.g. MySQL, but is dependant on the type of queries and the code specifics.

Do note, that you should run a few workers on your server, per the number of CPUs you have, for best performance. In regards to the 2nd Question: It is a much better practice, to name the functions, one after the other, and use the names in the code rather than defining it as you go. In some situations, it will reduce memory consumption.

Answer 2

So, every "user" should have their own redis client within the connection event. Am I right?

Actually, you are not :)

The thing is that node.js is very unlike, for example, PHP. node.js does not spawn child processes on new connections, which is one of the main reasons it can easily handle large amounts of concurrent connections, including long-lived connections (Comet, Websockets, etc.). node.js processes events sequentially using an event queue within one single process. If you want to use several processes to take advantage of multi-core servers or multiple servers, you will have to do it manually (how to do so is beyond the scope of this question, though).

Therefore, it is a perfectly valid strategy to use one single Redis (or MySQL) connection to serve a large quantity of clients. This avoids the overhead of instantiating and terminating a database connection for each client request.

Answer 3

So, every "user" should have their own redis client within the connection event. Am I right?

You shouldn't make a new Redis client for each connected user, that's not the proper way to do it. Instead just create 2-3 clients max and use them.

For more information checkout this question:

How to reuse redis connection in socket.io?