How does Stack Overflow generate its SEO-friendly

Question

What is a good complete regular expression or some other process that would take the title:

How do you change a title to be part of the URL like Stack Overflow?

and turn it into

how-do-you-change-a-title-to-be-part-of-the-url-like-stack-overflow

that is used in the SEO-friendly URLs on Stack Overflow?

The development environment I am using is Ruby on Rails, but if there are some other platform-specific solutions (.NET, PHP, Django), I would love to see those too.

I am sure I (or another reader) will come across the same problem on a different platform down the line.

I am using custom routes, and I mainly want to know how to alter the string to all special characters are removed, it's all lowercase, and all whitespace is replaced.

Answer 1

There is a small Ruby on Rails plugin called PermalinkFu, that does this. The escape method does the transformation into a string that is suitable for a URL. Have a look at the code; that method is quite simple.

To remove non-ASCII characters it uses the iconv lib to translate to 'ascii//ignore//translit' from 'utf-8'. Spaces are then turned into dashes, everything is downcased, etc.

Answer 2

T-SQL implementation, adapted from dbo.UrlEncode:

CREATE FUNCTION dbo.Slug(@string varchar(1024))
RETURNS varchar(3072)
AS
BEGIN
    DECLARE @count int, @c char(1), @i int, @slug varchar(3072)

    SET @string = replace(lower(ltrim(rtrim(@string))),' ','-')

    SET @count = Len(@string)
    SET @i = 1
    SET @slug = ''

    WHILE (@i <= @count)
    BEGIN
        SET @c = substring(@string, @i, 1)

        IF @c LIKE '[a-z0-9--]'
            SET @slug = @slug + @c

        SET @i = @i +1
    END

    RETURN @slug
END

Answer 3

Now all Browser handle nicely utf8 encoding, so you can use WebUtility.UrlEncode Method , its like HttpUtility.UrlEncode used by @giamin but its work outside of a web application.

Answer 4

For good measure, here's the PHP function in WordPress that does it... I'd think that WordPress is one of the more popular platforms that uses fancy links.

    function sanitize_title_with_dashes($title) {
            $title = strip_tags($title);
            // Preserve escaped octets.
            $title = preg_replace('|%([a-fA-F0-9][a-fA-F0-9])|', '---$1---', $title);
            // Remove percent signs that are not part of an octet.
            $title = str_replace('%', '', $title);
            // Restore octets.
            $title = preg_replace('|---([a-fA-F0-9][a-fA-F0-9])---|', '%$1', $title);
            $title = remove_accents($title);
            if (seems_utf8($title)) {
                    if (function_exists('mb_strtolower')) {
                            $title = mb_strtolower($title, 'UTF-8');
                    }
                    $title = utf8_uri_encode($title, 200);
            }
            $title = strtolower($title);
            $title = preg_replace('/&.+?;/', '', $title); // kill entities
            $title = preg_replace('/[^%a-z0-9 _-]/', '', $title);
            $title = preg_replace('/\s+/', '-', $title);
            $title = preg_replace('|-+|', '-', $title);
            $title = trim($title, '-');
            return $title;
    }

This function as well as some of the supporting functions can be found in wp-includes/formatting.php.

Answer 5

You can use the following helper method. It can convert the Unicode characters.

public static string ConvertTextToSlug(string s)
{
    StringBuilder sb = new StringBuilder();

    bool wasHyphen = true;

    foreach (char c in s)
    {
        if (char.IsLetterOrDigit(c))
        {
            sb.Append(char.ToLower(c));
            wasHyphen = false;
        }
        else
            if (char.IsWhiteSpace(c) && !wasHyphen)
            {
                sb.Append('-');
                wasHyphen = true;
            }
    }

    // Avoid trailing hyphens
    if (wasHyphen && sb.Length > 0)
        sb.Length--;

    return sb.ToString().Replace("--","-");
}

Answer 6

I ported the code to TypeScript. It can easily be adapted to JavaScript.

I am adding a .contains method to the String prototype, if you're targeting the latest browsers or ES6 you can use .includes instead.

if (!String.prototype.contains) {
    String.prototype.contains = function (check) {
        return this.indexOf(check, 0) !== -1;
    };
}

declare interface String {
    contains(check: string): boolean;
}

export function MakeUrlFriendly(title: string) {
            if (title == null || title == '')
                return '';

            const maxlen = 80;
            let len = title.length;
            let prevdash = false;
            let result = '';
            let c: string;
            let cc: number;
            let remapInternationalCharToAscii = function (c: string) {
                let s = c.toLowerCase();
                if ("àåáâäãåą".contains(s)) {
                    return "a";
                }
                else if ("èéêëę".contains(s)) {
                    return "e";
                }
                else if ("ìíîïı".contains(s)) {
                    return "i";
                }
                else if ("òóôõöøőð".contains(s)) {
                    return "o";
                }
                else if ("ùúûüŭů".contains(s)) {
                    return "u";
                }
                else if ("çćčĉ".contains(s)) {
                    return "c";
                }
                else if ("żźž".contains(s)) {
                    return "z";
                }
                else if ("śşšŝ".contains(s)) {
                    return "s";
                }
                else if ("ñń".contains(s)) {
                    return "n";
                }
                else if ("ýÿ".contains(s)) {
                    return "y";
                }
                else if ("ğĝ".contains(s)) {
                    return "g";
                }
                else if (c == 'ř') {
                    return "r";
                }
                else if (c == 'ł') {
                    return "l";
                }
                else if (c == 'đ') {
                    return "d";
                }
                else if (c == 'ß') {
                    return "ss";
                }
                else if (c == 'Þ') {
                    return "th";
                }
                else if (c == 'ĥ') {
                    return "h";
                }
                else if (c == 'ĵ') {
                    return "j";
                }
                else {
                    return "";
                }
            };

            for (let i = 0; i < len; i++) {
                c = title[i];
                cc = c.charCodeAt(0);

                if ((cc >= 97 /* a */ && cc <= 122 /* z */) || (cc >= 48 /* 0 */ && cc <= 57 /* 9 */)) {
                    result += c;
                    prevdash = false;
                }
                else if ((cc >= 65 && cc <= 90 /* A - Z */)) {
                    result += c.toLowerCase();
                    prevdash = false;
                }
                else if (c == ' ' || c == ',' || c == '.' || c == '/' || c == '\\' || c == '-' || c == '_' || c == '=') {
                    if (!prevdash && result.length > 0) {
                        result += '-';
                        prevdash = true;
                    }
                }
                else if (cc >= 128) {
                    let prevlen = result.length;
                    result += remapInternationalCharToAscii(c);
                    if (prevlen != result.length) prevdash = false;
                }
                if (i == maxlen) break;
            }

            if (prevdash)
                return result.substring(0, result.length - 1);
            else
                return result;
        }