Application level queueing / website traffic manag

问题:

We have a website application that we expect to receive incredibly high traffic at several points throughout the year. We currently have some third party load balancing software which redirects users to a 'holding' page during busy periods, to prevent our web application servers from being suffocated by the amount of requests coming in.

Going forward we would like to have more control over this process and implement a virtual queue of some kind. The current load balancer has no queuing functionality, but simply allows traffic through based on a rate limit. This is random and is pot luck on when you refresh the page (or get auto refreshed).

I've done some reading online about this but found little implementation detail on how to implement a very basic virtual HTTP request queue. There are of course companies that offer this as a fully fledged service such as queue-it and Netprecept but these seem overkill for our current needs (and are very expensive).

The web application in question is written in ASP.Net MVC. Bearing in mind that we do not need advanced features like 'queue priority' etc. at the moment, I have created a very basic proof-of-concept using a static queue manager class, using the ConcurrentQueue<T> etc. but am wondering if this is a valid, scalable approach? Is this something that can be part of the main application layer? Or should it be kept separate? Does any one have any technical know-how on how to implement this kind of feature into an ASP.Net MVC app?

---

EDIT: thanks for the answers so far. Most of the answers seem to go into a lot of detail about caching. This is already (very) heavily employed on our website, using ASP.Net web caching, caching full page requests at the load balancer level and object caching using AppFabric.

The reason for the ability to manage a queue is because the process is very database-write heavy. We're effectively creating orders for a certain product via the website. This means these DB transactions are taking things into account like last-minute stock checking etc. This is where the performance issues arise, and this is the reason for wanting to implement a queuing system of some kind.

Throwing more resources at the database server is not a realistic option. I'm really looking for details of technical implementations of a queuing system of this nature (C# or otherwise). Sorry if this wasn't made clear originally.

回答1:

Are you considering following points while Measuring the Performance of application?

1. Caching
2. Sessionless Controllers
3. AsyncControllers

Output caching :

Perhaps the most useful feature of MVC3 (Performance-Wise) is output caching. The biggest Performance hits actually occur when your application really has to fetch data, do calculations on it and return the data. Output caching can cache these results so they can be returned directly without even touching the database. Especially when executing complex queries this can drop the load on your server significantly (in fact you could drop the load on your server by a whooping 90% by carefully inplementing caching in your web application).

namespace MvcApplication1.Controllers
{
     public class DataController : Controller
     {
          [OutputCache(Duration=10)]
          public string Index()
          {
               return DateTime.Now.ToString("T");    
          }
     }
}

Sessionless controllers :

Controllers with session state disabled provide an optimization for controllers that do not require session state. Stateless controllers are meant for situations where you do not require the concept of a session.

By default the ASP.NET pipeline will not process requests belonging to the same session concurrently. It serialises them, i.e. it queues them in the order that they were received so that they are processed serially rather than in parallel. This means that if a request is in progress and another request from the same session arrives, it will be queued to only begin executing when the first request has finished.

Let's look at an example; a page making 3 asynchronous AJAX requests to the server, with session state enabled(also note that session must actually be used, as ASP.NET is smart enough not to serialise requests if you never use session state, even if it's enabled).

JQuery

$(document).ready(function () {
    //Make 3 concurrent requests to /ajaxtest/test
    for (var i = 0; i < 3; i++) {       
        $.post("/ajaxtest/test/" + i,
            function (data) {                       
                //Do something with data...
            },
            "json");
    }
});

Controller - Action Method

public class AjaxTestController : Controller
{       
    [HttpPost]
    public JsonResult Test(int? id)
    {
        Thread.Sleep(500);
        return Json(/*Some object*/);
    }
}

You can see the effect of serialised requests in the network profile; each request takes roughly 500ms longer than the previous one. So it means we're not getting any benefit from making these AJAX calls asynchronously. Let's look at the profile again with session state disabled for our AjaxTestController (using the [SessionState] attribute).

[SessionState(SessionStateBehavior.Disabled)]
public class AjaxTestController : Controller
{       
    //...As above
}

Much better! You can see how the 3 requests are being processed in parallel, and take a total of 500ms to complete, rather than 1500ms we saw in our first example.

Async-Controllers :

First, controller begins one or more external I/O calls (e.g., SQL database calls or web service calls). Without waiting for them to complete, it releases the thread back into the ASP.NET worker thread pool so that it can deal with other requests.

Later, when all of the external I/O calls have completed, the underlying ASP.NET platform grabs another free worker thread from the pool, reattaches it to your original HTTP context, and lets it complete handling the original request.

How to Measure the Response time under Heavy Traffic?

I copied below content from this link. Because sometime links gets broken so I kept some important part here. Please check this link for more details

To understand how asynchronous controllers respond to differing levels of traffic, and how this compares to a straightforward synchronous controller, you can put create a sample MVC with two controllers. To simulate a long-running external, they both perform a SQL query that takes 2 seconds to complete (using the SQL command WAITFOR DELAY ’00:00:02′) and then they return the same fixed text to the browser. One of them does it Synchronously; the other Asynchronously.

In another example you can check a simple C# console application that simulates heavy traffic hitting a given URL. It simply requests the same URL over and over, calculating the rolling average of the last few response times. First it does so on just one thread, but then gradually increases the number of concurrent threads to 150 over a 30-minute period. If you want to try running this tool against your own site, you can download the C# source code.

The results illustrate a number of points about how asynchronous requests perform. Check out this graph of average response times versus number of concurrent requests (lower response times are better):

To understand this, first I need to tell you that I had set my ASP.NET MVC application’s worker thread pool to an artificially low maximum limit of 50 worker threads. My server actually has a default max threadpool size of 200 – a more sensible limit – but the results are made clearer if I reduce it. As you can see, the synchronous and asynchronous requests performed exactly the same as long as there were enough worker threads to go around. And why shouldn’t they? But once the threadpool was exhausted (> 50 clients), the synchronous requests had to form a queue to be serviced. Basic queuing theory tells us that the average time spent waiting in a queue is given by the formula:

and this is exactly what we see in the graph. The queuing time grows linearly with the length of the queue. (Apologies for my indulgence in using a formula – sometimes I just can’t suppress my inner mathematician. I’ll get therapy if it becomes a problem.) The asynchronous requests didn’t need to start queuing so soon, though. They don’t need to block a worker thread while waiting, so the threadpool limit wasn’t an issue. So why did they start queuing when there were more than 100 clients? It’s because the ADO.NET connection pool is limited to 100 concurrent connections by default.

Hope this should help you.

回答2:

IIS has it's own queues which you can configure for each application pool. Here is the link containing usefull tips on performance here.

I wouldn't recomment mixing application code and code which deals with low technical performance optimisations. Keep it separate as it will keep you code mantainable.

回答3:

Depending on your application queueing might not solve your problem at all. If you build up a queue your server will have less work to do at the same time because less concurrent threads are running.

When to queue?

Obviously i've to reverse the question: Why are you queuing? Propably because your servers just can't handle the amounts of incoming traffic. If we're talking about peaks which last only for a few seconds this is not a big deal: Build up a queue and process it in a few seconds.

A queue is just a bucket

As your question states this peaks last longer than a few seconds "several points throughout the year". So i assume that your servers are experiencing massive inrush of new requests for a time which is long enough that processing the queue would just result in timeouts on the clients. [FYI: Check out the bucket model in networking. These has the same issues to deal with]

Let me give an easy example:

It's a normal day in summer and you're offering cold lemonade on a street corner. You can handle up to 5 clients per minute and the peaks during lunch mean that about 7 clients are queuing up for your lemonade per minute. After one minute you've 2 clients waiting, after two minutes you have 4 clients waiting for you to process their orders [and so on].

As the peak inrush just holds for let's say 10 minutes you're queue just reaches a maximum length of 20 clients.

On another day it's very hot and your inrush peaks are not the usual 7 clients per minute, your inrush peaks up to 42 clients per minute. This means that already after one minute you've 37 clients waiting for your lemonade. It's obviously that - except it's the last lemonade on earth - your clients won't queue an hour for a cup of lemonade and will just leave.

And this is the same problem concerning your peaks on the server: If you're continously experiencing more incoming connections than you can handle your queue will grow until it's full and then drop incoming clients. So all you're gaining with queuing is that you just delay the first dropped clients and put other clients on hold (which is also annoying).

Going back the the real world example. How can this problem be solved here? It's quiet easy: speed up processing each client or get some clerks and open up a second, third,... queue to handle multiple clients at once and process all orders. As in real world and in programming: speeding up is not that easy. getting up multiple instances just for short periods of time is also quiet difficult. But clearly spoken: these are the only reliable solutions to handle such problems.

Using IIS and MVC: Not really queuing at the right place

[Queuing up on a single queue] You may build up a queue in your application and suspend all inrushing requests if you've reached your maximum concurrent active workers. But this doesn't mean that there is no overhead for all the inrushing tcp connections and asp sessions which are spawned and loaded on each request as IIS accepts new connections if there are free concurrent connections from the IIS' viewpoint.

Building a http proxy application

[Queuing up on a single or multiple queues] If you're building a custom proxy application you may be able decide yourself when you want to accept new incoming client connections (suspending the tcp listener or just not accepting new clients until your workload has enough resources to handle the new clients). Of course your proxy could spread the requests to several hosts and do some sort of load balancing - but this requires that you have additional hosts you can instantly fire up and handle the requests (this is what any loadbalancer would require from you: have a backend which may handle the workload). As this is an absolutely low level apporach i think it's not required to go into further detail and explain that there are plenty of risks when taking this road.

Concerning some simple aspects to just reduce some parts of the workload: - Offload static contents like images to other servers / sites which are low cost. This reduces the workload on your primary server which do the hard, dynamic work. - Caching: Don't do things twice which have the same results. Check where you're application gets slow in high workloads and try to optimize them to be as fast as possible. - Work async / put long runners into sleep: If you've external requests (database) try to implement them in an async manner to the your threads be running as long as possible and then go to sleep to give the scheduler of your host the chance to process other pending work. Benchmark your threads and check how long the inclusive times of your critical methods are and where they are spending their time.

Hope this helps to solve your problem or just get's you an idea what road to take.

回答4:

I would recommend that you use something like BIG-IP , if you are going to use a Software based approach then on what basis will you Redirect your request to which Servers? There are lot of choices like Round Robin..etc

Take a look at this article if it helps

回答5:

You could have a look at getting some benchmark figures to see what kind of load/traffic your current setup (infrastructure and appplication) can handle e.g. how many concurrent users etc.

You could do this possibly using Microsoft Team Foundation Service (hosted in the cloud, different to TF Server which is hosted on premise i.e. your infrastructure. using the Load/Stress/Performance tests

I believe TFS Service is free for a dev team smaller than 5 developers I think. Dont quote me on that :-)

Once you get some ball park figures and if your application can cope then nothing to do. If it cant then some decisions to make? Your then in the 80/20 scenario. Is it worth spending 80% of your resource (dev time, wages etc) on a possible 20% performance gain or is it better to invest 20% hardware cost (CPU, RAM, another server etc) for an instant 80% performance boost?

If this site is high profile for your business or customer and is generating income or has legislative purpose then you may find an invest in hardware easier then investment in software development. That way you can look to have an high-available (HA) scalable application without a single point of failure (SPF).

Not really answering your queueing questions but just another viewpoint to consider.

FYI we are hoping to upgrade to two VMs (running VMWare) front end webservers running windows server 2012 enterprise edition. These will be network load balanced using microsoft NLB (so no need for another load balancing server or third party software). We will have an active-active cluster for the backedn SQL servers. this allows us to maintain the servers (upgrades, patches etc) without having downtime. AN additional benefit is with IIS 8.0 you can run multiple domain (hosts) of the same IP for SSL (HTTPS) traffic using SNI which is nice (but torpedos user running windows xp but its out of support in April 2014 anyway). We have to upgradeour .net applications to take care of session management and MACstate(cross server postbacks etc) but the trade of is worth it.

回答6:

Along with PKKG's answer, you will have to redesign certain parts of your application to add caching.

ASP.NET MVC output cache supports SQL cache dependency which lets you cache the page by both server and client based on some version column in table.

Let's assume you have a page that gets displayed very frequently but changes less frequently. For example, Question page of stack Overflow. Each question has probably 1:100 write vs read operations. This means for every read, connecting to DB is very high cost.

So what we can do is, treat LostUpdate column as Version column and add SQL Cache Dependency on this page with SQL query which returns true If LastUpdate for current question is later then the given cached time.

For every single page, Cache manager does make cache query to SQL, but this query is smaller then entire page, because page contains lots of query to different table as well to build and show every information. Such as, answers, and their authors and their reputations etc. So for 100 times, only one query to LostUpdate saves huge overhead against 100 times all multiple queries & joins & many other calculations.

Doing this on each page is difficult but you can analyze traffic pattern and improve caching. And in same example with Stake Overflow, caching on home page may be bad idea as it displays recent updated entries, so caching may not work, as page changes too quickly.

回答7:

Some options are use sql service broker, or a replication snapshot. With snapshot you can move the reporting process to other copy of the database and keep the locks low.