How can I detect and survive being “Slashdotted”?

Question

What's a good way to survive abnormally high traffic spikes?

My thought is that at some trigger, my website should temporarily switch into a "low bandwidth" mode: switch to basic HTML pages, minimal graphics, disable widgets that might put unnecessary load on the database, and so-on.

My thoughts are:

• Monitor CPU usage
• Monitor bandwidth
• Monitor requests / minute

Edit: I am familiar with options like caching, switching to static content or a content delivery network, and so on as a means to survive, so perhaps the question should focus more on how one detects when the website is about to become overloaded. (Although answers on other survival methods are of course still more than welcome.) Lets say that the website is running Apache on Linux and PHP. This is probably the most common configuration and should allow the maximum number of people to gain assistance from the answers. Lets also assume that expensive options like buying another server and load balancing are unavailable - for most of us at least, a mention on Slashdot is going to be a once-in-a-lifetime occurrence, and not something we can spend money preparing for.

Answer 1

Here's a rather lengthy but highly informative article about surviving "flash crowds".

Here's their scenario for the situation their proposed solutions address:

In this paper, we consider the question of scaling through the eyes of a character we call the garage innovator. The garage innovator is creative, technically savvy, and ambitious. She has a great idea for the Next Big Thing on the web and implements it using some spare servers sitting out in the garage. The service is up and running, draws new visitors from time to time, and makes some meager income from advertising and subscriptions. Someday, perhaps, her site will hit the jackpot. Maybe it will reach the front page of Slashdot or Digg; maybe Valleywag or the New York Times will mention it.

Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. Since her idea is so novel, many will become revenue-generating customers and refer friends. But a flash crowd is notoriously fickle; the outcome won't be nearly as idyllic if the site crashes under its load. Many people won't bother to return if the site doesn't work the first time. Still, it is hard to justify paying tens of thousands of dollars for resources just in case the site experiences a sudden load spike. Flash crowds are both the garage innovator's bane and her goal.

One way out of this conundrum has been enabled by contemporary utility computing.

The article then proposed a number of steps the garage innovator can take, such as using storage delivery networks and implementing highly-scalable databases.

Answer 2

nearlyfreespeech.net is a semi-cloud so to speak and helps a ton in situations like this. As others above mentioned, layered caching helps a lot. Pull chunks of information from memcached instead of the database, have a reverse proxy (or a distributed reverse proxy aka CDN, Panther Networks is cheap) in front of you.

Answer 3

Auto-redirect to Coral CDN, unless the request is from coral cdn.

Answer 4

.htaccess:

RewriteEngine on
RewriteCond %{HTTP_REFERER} slashdot\.org [NC]
RewriteRule .* - [F]

Answer 5

I rewrite all URLs referred by several popular sites to be redirected through the coralCDN.

An example for Apache:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} !^Googlebot
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?digg\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?slashdot\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?slashdot\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?fark\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?somethingawful\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?kuro5hin\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?engadget\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?boingboing\.net [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?del\.icio\.us [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?delicious\.com
RewriteRule ^(.*)?$ http://example.com.nyud.net/$1 [R,L]
</IfModule>

Answer 6

Don't write content or provide a service that may appeal to geeks ;)