How to link to a child site-map file from a parent

I am using MVCSitemapProvider by Maarten Balliauw with Ninject DI in MVC4. Being a large-scale web app, enumerating over the records to generate the sitemap xml accounts for 70% of the page load time. For that purpose, I went for using new sitemap files for each level-n dynamic node provider.

<mvcSiteMap xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://mvcsitemap.codeplex.com/schemas/MvcSiteMap-File-4.0" xsi:schemaLocation="http://mvcsitemap.codeplex.com/schemas/MvcSiteMap-File-4.0 MvcSiteMapSchema.xsd">
    <mvcSiteMapNode title="$resources:SiteMapLocalizations,HomeTitle" description="$resources:SiteMapLocalizations,HomeDescription" controller="Controller1" action="Home" changeFrequency="Always" updatePriority="Normal" metaRobotsValues="index follow noodp noydir"><mvcSiteMapNode title="$resources:SiteMapLocalizations,AboutTitle" controller="ConsumerWeb" action="Aboutus"/>
    <mvcSiteMapNode title="Sitemap" controller="Consumer1" action="SiteMap"/><mvcSiteMapNode title=" " action="Action3" controller="Consumer2" dynamicNodeProvider="Comp.Controller.Utility.NinjectModules.PeopleBySpecDynamicNodeProvider, Comp.Controller.Utility" />
    <mvcSiteMapNode title="" siteMapFile="~/Mvc2.sitemap"/>
    </mvcSiteMapNode>
    </mvcSiteMap>

But, it doesn't seem to work. For localhost:XXXX/sitemap.xml, the child nodes from Mvc2.sitemap don't seem to load.

siteMapFile is not a valid XML attribute in MvcSiteMapProvider (although you could use it as a custom attribute), so I am not sure what guide you are following to do this. But, the bottom line is there is no feature that loads "child sitemap files", and even if there was, it wouldn't help with your issue because all of the nodes are loaded into memory at once. Realistically on an average server there is an upper limit of around 10,000 - 15,000 nodes.

The problem that you describe is a known issue. There are some tips available in issue #258 that may or may not help.

We are working a new XML sitemap implementation that will allow you to connect the XML sitemap directly to your data source, which can be used to circumvent this problem (at least as far as the XML sitemap is concerned). This implementation is stream-based and has paging that can be tied directly to the data source, and will seamlessly page over multiple tables, so it is very efficient. However, although there is a working prototype, it is still some time off from being made into a release.

If you need it sooner rather than later, you are welcome to grab the prototype from this branch.

You will need some code to wire it into your application (this is subject to change for the official release). I have created a demo project here.

Application_Start

var registrar = new MvcSiteMapProvider.Web.Routing.XmlSitemapFeedRouteRegistrar();
registrar.RegisterRoutes(RouteTable.Routes, "XmlSitemap2");

XmlSitemap2Controller

using MvcSiteMapProvider.IO;
using MvcSiteMapProvider.Web.Mvc;
using MvcSiteMapProvider.Xml.Sitemap.Configuration;
using System.Web.Mvc;

public class XmlSitemap2Controller : Controller
{
    private readonly IXmlSitemapFeedResultFactory xmlSitemapFeedResultFactory;

    public XmlSitemap2Controller()
    {
        var builder = new XmlSitemapFeedStrategyBuilder();

        var xmlSitemapFeedStrategy = builder
            .SetupXmlSitemapProviderScan(scan => scan.IncludeAssembly(this.GetType().Assembly))
            .AddNamedFeed("default", feed => feed.WithMaximumPageSize(5000).WithContent(content => content.Image().Video()))
            .Create();

        var outputCompressor = new HttpResponseStreamCompressor();

        this.xmlSitemapFeedResultFactory = new XmlSitemapFeedResultFactory(xmlSitemapFeedStrategy, outputCompressor);
    }

    public ActionResult Index(int page = 0, string feedName = "")
    {
        var name = string.IsNullOrEmpty(feedName) ? "default" : feedName;
        return this.xmlSitemapFeedResultFactory.Create(page, name);
    }
}

IXmlSiteMapProvider

And you will need 1 or more IXmlSitemapProvider implementations. For convenience, there is a base class XmlSiteMapProviderBase. These are similar to creating controllers in MVC.

using MvcSiteMapProvider.Xml.Sitemap;
using MvcSiteMapProvider.Xml.Sitemap.Specialized;
using System;
using System.Linq;

public class CategoriesXmlSitemapProvider : XmlSitemapProviderBase, IDisposable
{
    private EntityFramework.MyEntityContext db = new EntityFramework.MyEntityContext();

    // This is optional. Don't override it if you don't want to use last modified date.
    public override DateTime GetLastModifiedDate(string feedName, int skip, int take)
    {
        // Get the latest date in the specified page
        return db.Category.OrderBy(x => x.Id).Skip(skip).Take(take).Max(c => c.LastUpdated);
    }

    public override int GetTotalRecordCount(string feedName)
    {
        // Get the total record count for all pages
        return db.Category.Count();
    }

    public override void GetUrlEntries(IUrlEntryHelper helper)
    {
        // Do not call ToList() on the query. The idea is that we want to force
        // EntityFramework to use a DataReader rather than loading all of the data
        // at once into RAM.
        var categories = db.Category
            .OrderBy(x => x.Id)
            .Skip(helper.Skip)
            .Take(helper.Take);

        foreach (var category in categories)
        {
            var entry = helper.BuildUrlEntry(string.Format("~/Category/{0}", category.Id))
                .WithLastModifiedDate(category.LastUpdated)
                .WithChangeFrequency(MvcSiteMapProvider.ChangeFrequency.Daily)
                .AddContent(content => content.Image(string.Format("~/images/category-image-{0}.jpg", category.Id)).WithCaption(category.Name));

            helper.SendUrlEntry(entry);
        }
    }

    public void Dispose()
    {
        db.Dispose();
    }
}

Note that there is currently not an IXmlSiteMapProvider implementation that reads the nodes from the default (or any) SiteMap, but creating one is similar to what is shown above, except you would query the SiteMap for nodes instead of a database for records.

Alternatively, you could use a 3rd party XML sitemap generator. Although, nearly all of them are set up in a non-scalable way for large sites, and most leave it up to you to handle the paging. If they aren't streaming the nodes, it will not realistically scale to more than a few thousand URLs.

The other detail you might need to take care of is to use the forcing a match technique to reduce the total number of nodes in the SiteMap. If you are using the Menu and/or SiteMap HTML helpers, you will need to leave all of your high-level nodes alone. But any node that does not appear in either is a good candidate for this. Realistically, nearly any data-driven site can be reduced to a few dozen nodes using this technique, but keep in mind every node that is forced to match multiple routes in the SiteMap means that individual URL entries will need to be added in the XML sitemap.