Tips for debugging .htaccess rewrite rules

Question

Many posters have problems debugging their RewriteRule and RewriteCond statements within their .htaccess files. Most of these are using a shared hosting service and therefore don't have access to the root server configuration. They cannot avoid using .htaccess files for rewriting and cannot enable a RewriteLogLevel" as many respondents suggest. Also there are many .htaccess-specific pitfalls and constraints are aren't covered well. Setting up a local test LAMP stack involves too much of a learning curve for most.

So my Q here is how would we recommend that they debug their rules themselves. I provide a few suggestions below. Other suggestions would be appreciated.

1. Understand that the mod_rewrite engine cycles through .htaccess files. The engine runs this loop:

   do
     execute server and vhost rewrites (in the Apache Virtual Host Config)
     find the lowest "Per Dir" .htaccess file on the file path with rewrites enabled
     if found(.htaccess)
        execute .htaccess rewrites (in the user's directory)
   while rewrite occurred
   

   So your rules will get executed repeatedly and if you change the URI path then it may end up executing other .htaccessfiles if they exist. So make sure that you terminate this loop, if necessary by adding extra RewriteCond to stop rules firing. Also delete any lower level .htaccess rewrite rulesets unless explicitly intent to use multi-level rulesets.

2. Make sure that the syntax of each Regexp is correct by testing against a set of test patterns to make sure that is a valid syntax and does what you intend with a fully range of test URIs. See answer below for more details.

3. Build up your rules incrementally in a test directory. You can make use of the "execute the deepest .htaccess file on the path feature" to set up a separate test directory (tree) and debug rulesets here without screwing up your main rules and stopping your site working. You have to add them one at a time because this is the only way to localise failures to individual rules.

4. Use a dummy script stub to dump out server and environment variables. (See Listing 2)If your app uses, say, blog/index.php then you can copy this into test/blog/index.php and use it to test out your blog rules in the test subdirectory. You can also use environment variables to make sure that the rewrite engine in interpreting substitution strings correctly, e.g.

   RewriteRule ^(.*) - [E=TEST0:%{DOCUMENT_ROOT}/blog/html_cache/$1.html]
   

   and look for these REDIRECT_* variables in the phpinfo dump. BTW, I used this one and discovered on my site that I had to use %{ENV:DOCUMENT_ROOT_REAL} instead. In the case of redirector looping REDIRECT_REDIRECT_* variables list the previous pass. Etc..

5. Make sure that you don't get bitten by your browser caching incorrect 301 redirects. See answer below. My thanks to Ulrich Palha for this.

6. The rewrite engine seems sensitive to cascaded rules within an .htaccess context, (that is where a RewriteRule results in a substitution and this falls though to further rules), as I found bugs with internal sub-requests (1), and incorrect PATH_INFO processing which can often be prevents by use of the [NS], [L] and [PT] flags.

Any more comment or suggestions?

Listing 1 -- phpinfo

<?php phpinfo(INFO_ENVIRONMENT|INFO_VARIABLES);

Answer 1

I'll leave this here, maybe obvious detail, but got me banging my head for hours: be careful using %{REQUEST_URI} because what @Krist van Besien say in his answer is totally right, but not for the REQUEST_URI string, because the out put of this TestString starts with a /. So take care:

RewriteCond %{REQUEST_URI} ^/assets/$  
                            ^
                            | check this pesky fella right here if missing

Answer 2

Make sure that the syntax of each Regexp is correct

by testing against a set of test patterns to make sure that is a valid syntax and does what you intend with a fully range of test URIs.

See regexpCheck.php below for a simple script that you can add to a private/test directory in your site to help you do this. I've kept this brief rather than pretty. Just past this into a file regexpCheck.php in a test directory to use it on your website. This will help you build up any regexp and test it against a list of test cases as you do so. I am using the PHP PCRE engine here, but having had a look at the Apache source, this is basically identical to the one used in Apache. There are many HowTos and tutorials which provide templates and can help you build your regexp skills.

Listing 1 -- regexpCheck.php

<html><head><title>Regexp checker</title></head><body>
<?php 
    $a_pattern= isset($_POST['pattern']) ? $_POST['pattern'] : "";
    $a_ntests = isset($_POST['ntests']) ? $_POST['ntests'] : 1;
    $a_test   = isset($_POST['test']) ? $_POST['test'] : array();

    $res = array(); $maxM=-1; 
    foreach($a_test as $t ){
        $rtn = @preg_match('#'.$a_pattern.'#',$t,$m);
        if($rtn == 1){
            $maxM=max($maxM,count($m));
            $res[]=array_merge( array('matched'),  $m );
        } else {
            $res[]=array(($rtn === FALSE ? 'invalid' : 'non-matched'));
        }
    } 
?> <p>&nbsp; </p>
<form method="post" action="<?php echo $_SERVER['SCRIPT_NAME'];?>">
    <label for="pl">Regexp Pattern: </label>
    <input id="p" name="pattern" size="50" value="<?php echo htmlentities($a_pattern,ENT_QUOTES,"UTF-8");;?>" />
    <label for="n">&nbsp; &nbsp; Number of test vectors: </label>
    <input id="n" name="ntests"  size="3" value="<?php echo $a_ntests;?>"/>
    <input type="submit" name="go" value="OK"/><hr/><p>&nbsp;</p>
    <table><thead><tr><td><b>Test Vector</b></td><td>&nbsp; &nbsp; <b>Result</b></td>
<?php 
    for ( $i=0; $i<$maxM; $i++ ) echo "<td>&nbsp; &nbsp; <b>\$$i</b></td>";
    echo "</tr><tbody>\n";
    for( $i=0; $i<$a_ntests; $i++ ){
        echo '<tr><td>&nbsp;<input name="test[]" value="', 
            htmlentities($a_test[$i], ENT_QUOTES,"UTF-8"),'" /></td>';
        foreach ($res[$i] as $v) { echo '<td>&nbsp; &nbsp; ',htmlentities($v, ENT_QUOTES,"UTF-8"),'&nbsp; &nbsp; </td>';}
        echo "</tr>\n";
    }
?> </table></form></body></html>

Answer 3

Regarding 4., you still need to ensure that your "dummy script stub" is actually the target URL after all the rewriting is done, or you won't see anything!

A similar/related trick (see this question) is to insert a temporary rule such as:

RewriteRule (.*) /show.php?url=$1 [END]

Where show.php is some very simple script that just displays its $_GET parameters (you can display environment variables too, if you want).

This will stop the rewriting at the point you insert it into the ruleset, rather like a breakpoint in a debugger.

If you're using Apache <2.3.9, you'll need to use [L] rather than [END], and you may then need to add:

RewriteRule ^show.php$ - [L]

At the very top of your ruleset, if the URL /show.php is itself being rewritten.

Answer 4

Here are a few additional tips on testing rules that may ease the debugging for users on shared hosting

1. Use a Fake-user agent

When testing a new rule, add a condition to only execute it with a fake user-agent that you will use for your requests. This way it will not affect anyone else on your site.

e.g

#protect with a fake user agent
RewriteCond %{HTTP_USER_AGENT}  ^my-fake-user-agent$
#Here is the actual rule I am testing
RewriteCond %{HTTP_HOST} !^www\.domain\.com$ [NC] 
RewriteRule ^ http://www.domain.com%{REQUEST_URI} [L,R=302] 

If you are using Firefox, you can use the User Agent Switcher to create the fake user agent string and test.

2. Do not use 301 until you are done testing

I have seen so many posts where people are still testing their rules and they are using 301's. DON'T.

If you are not using suggestion 1 on your site, not only you, but anyone visiting your site at the time will be affected by the 301.

Remember that they are permanent, and aggressively cached by your browser. Use a 302 instead till you are sure, then change it to a 301.

3. Remember that 301's are aggressively cached in your browser

If your rule does not work and it looks right to you, and you were not using suggestions 1 and 2, then re-test after clearing your browser cache or while in private browsing.

4. Use a HTTP Capture tool

Use a HTTP capture tool like Fiddler to see the actual HTTP traffic between your browser and the server.

While others might say that your site does not look right, you could instead see and report that all of the images, css and js are returning 404 errors, quickly narrowing down the problem.

While others will report that you started at URL A and ended at URL C, you will be able to see that they started at URL A, were 302 redirected to URL B and 301 redirected to URL C. Even if URL C was the ultimate goal, you will know that this is bad for SEO and needs to be fixed.

You will be able to see cache headers that were set on the server side, replay requests, modify request headers to test ....

---

Answer 5

Set environment variables and use headers to receive them:

You can create new environment variables with RewriteRule lines, as mentioned by OP:

RewriteRule ^(.*) - [E=TEST0:%{DOCUMENT_ROOT}/blog/html_cache/$1.html]

But if you can't get a server-side script to work, how can you then read this environment variable? One solution is to set a header:

Header set TEST_FOOBAR "%{REDIRECT_TEST0}e"

The value accepts format specifiers, including the %{NAME}e specifier for environment variables (don't forget the lowercase e). Sometimes, you'll need to add the REDIRECT_ prefix, but I haven't worked out when the prefix gets added and when it doesn't.

Answer 6

Online .htaccess rewrite testing

I found this Googling for RegEx help, it saved me a lot of time from having to upload new .htaccess files every time I make a small modification.

from the site:

htaccess tester

To test your htaccess rewrite rules, simply fill in the url that you're applying the rules to, place the contents of your htaccess on the larger input area and press "Check Now" button.