Azure Web Jobs and SignalR memory leak

Question

So I have some webjobs that occasionally connect to a signalr hub and broadcast a message. Below is just an example of one, in this case its a simple web job for development with the TimerTrigger attribute that is set to run continuously every 20 seconds. Like that shown in the code below.

    public static void Main()
    {

        JobHostConfiguration config = new JobHostConfiguration();
        config.Tracing.ConsoleLevel = TraceLevel.Verbose;
        config.UseTimers();
        if (config.IsDevelopment)
        {
            config.UseDevelopmentSettings();
        }
        var host = new JobHost(config);
        host.RunAndBlock();
    }

    public static void ProcessPush([TimerTrigger("00:00:20", RunOnStartup = true)] TimerInfo timerInfo, TextWriter log)
    {
        // Send a signalr message to the Hub
        try
        {            
            SendMessageToHub(log);
        }
        catch (Exception e)
        {
            log.WriteLine($"WebJob Push Exception: {e.Message}");
        }
    }

    private static async Task SendMessageToHub(TextWriter log)
    {
            var hub = new HubConnection(CloudConfigurationManager.GetSetting("MyWebSite"));
            var proxy = _hub.CreateHubProxy("MyHub");

            log.WriteLine("WebJob Push: Sending message to SignalR Hub.");
            if (_hub.State == Microsoft.AspNet.SignalR.Client.ConnectionState.Disconnected)
            {
                await _hub.Start();
            }
            await _proxy.Invoke("BroadcastMessage");
            log.WriteLine("WebJob Push: Sent message to SignalR Hub.");
    }

There is always an increment in memory on the server hosting the website and signalr hubs. When investigating the IIS logs on the website, it seems to have surge / batch of POST messages come to the website at the same second, using long polling. It then waits a little while and then gets bombarded with another batch of messages. By the way, this drives the CPU on the IIS Server crazy as well. At the bottom of this post is an example of the IIS log entries.

I'd like to be able to send a signalr message from the web job in a consistent manner with a regular pulse message (we want to expand on the web job - please excuse the fact it runs on a timer for now).

Kind regards,

Stefan

Example IIS Log entries - notice they have all come in within the same second, instead of 20 seconds apart:

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=TBbNVDpndk0riu8UvVzbGJrWjYIo7eMLcP4lk7ABV74OBMbZRTJrCRL1bzsPxpd1Tyle2rS3tV2JJrigninhu880ml51Xers76PPDX0Hf97dTBYR4k%2BVc2V9KAmiGt0p&messageId=d-80E0087-B%2C7D%7CEz%2C0%7CE0%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 5343

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=KYwQXpNrPIU21NXMa0So5u42EwXTcMlGyLqL3tetx4WfOtTunHLclG%2BhPd%2BcPeZPmfe6KKvQL13XIU1W5fApuTv0XN5XFPoNUmyBjhhISoqodwcZeu3QKmkbaXcpHMtE&messageId=d-80E0087-B%2C7D%7Cav%2C0%7Caw%2C2 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 10641

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=nO%2BPZ8M5JJOpiobpJUV5%2FZvQyEKYjp%2FOuqQ%2F0Bkq05TKRJZfeI%2FD%2BxRyPC7EsAAjXVqJr05PksorlMWrXocGkskfVsLU2Qvtx%2Fi1O8hU5lNz4KcoSc%2Bkv%2BlDpr2AZBLv&messageId=d-80E0087-B%2C7D%7CFB%2C0%7CFC%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 18282

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=wiGSRiNHdd7crhkcAMd%2FWy%2F3qGRZ5WdBm%2BdbR3b7aTbtpB8aaBGDil%2FqAWha6Si5eEohsUmCxAU4Pkefy%2BNoxoG9fgYC4R66ErXIShyBUcsNLWo1AyH5zGDk7bFvme3E&messageId=d-80E0087-B%2C7D%7CE9%2C0%7CE_%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 11360

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=hEJ1b0%2Bz2eeyC8IvYmOV3ffZ%2FAFQiQpEnJLUmCZTEVDLwcgOqhyQbQnu0R29sazp6BxcK4WsDhSbEdg2Sh4wMBSZjQtKMzASr2Fa2eY2HGgoVJcfDOMixQX2FCqfa%2BmP&messageId=d-80E0087-B%2C7D%7CFD%2C0%7CFE%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 15798

2016-11-15 23:10:35 POST /signalr/poll clientProtocol=1.4&transport=longPolling&connectionData=[%7B%22Name%22:%22MyHub%22%7D]&connectionToken=2UsU63IHgaNO%2BBYmoamsKxFq7Vv3uaGigvR1NrGnntVnAbTg2C0%2BVXZnA9aT8siqpkBv%2Fo8avvvNTSBfQD77IspaO6jOnSU8rXMXDU2Vr6ojkWr%2Fwt1LFsdNy3%2BHpDGC&messageId=d-80E0087-B%2C7D%7CEv%2C0%7CEw%2C0 443 - 104.210.116.149 SignalR.Client.NET45/2.2.1.0+(Microsoft+Windows+NT+6.2.9200.0) - 200 0 0 11844

etc, etc

UPDATE - Explicitly stopping the hub connection seems to have dealt with the orphan clients (or superfluous clients for the same web job client). In particular, adding _hub.Stop(); after invoking the proxy.

Answer 1

So as obvious as it may sound to some, the answer to this was to ensure that we stop the hub connection after sending the message to it. Below is the code with the additional line surrounded by comments.

It would seem that unlike a browser client, whereby the client gets disconnected after a while of inactivity. The webjob client sticks around, and continues to poll the website. So each time the webjob is triggered (via a timer or via reading a message off an azure queue or topic) then it keeps spawning a new client and retaining that connection.

In a webjob, you must explicitly stop the connection, otherwise the memory and CPU will gradually go nutbags.

private static async Task SendMessageToHub(TextWriter log)
{
        var hub = new HubConnection(CloudConfigurationManager.GetSetting("MyWebSite"));
        var proxy = _hub.CreateHubProxy("MyHub");

        log.WriteLine("WebJob Push: Sending message to SignalR Hub.");
        if (_hub.State == Microsoft.AspNet.SignalR.Client.ConnectionState.Disconnected)
        {
            await _hub.Start();
        }
        await _proxy.Invoke("BroadcastMessage");
        /////////////////////////////////////////////////////////////
        // Stopping the hub connection is necesssary in a web job  //
        _hub.Stop();   
        /////////////////////////////////////////////////////////////
        log.WriteLine("WebJob Push: Sent message to SignalR Hub.");
}