HttpBrowserCapabilities.Crawler property .NET

Question

How does the HttpBrowserCapabilities.Crawler property (http://msdn.microsoft.com/en-us/library/aa332775(VS.71).aspx) work? I need to detect a partner's custom crawler and this property is returning false.

Where/How can I add his user agent so that this property will return true? Any other way outside of creating my own user agent detecting mechanism?

Answer 1

This is all driven by the default browsercaps declarations that are part of the .NET framework.

To setup this specific crawler, you would need to add a custom browserCaps section to your web.config to identify it as a crawler.

The MSDN documentation on how to go about this is pretty well documented, here is a link.

Answer 2

HttpBrowserCapabilities is populated from a set of *.browser files in C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG\Browsers. You can add your own file in ~/App_Browsers/ with the details of your custom crawler. The schema is here. Basically it contains a regular expression to match against the User-Agent header and a collection of property values.