AWS S3 redirect not working

I've just created a new bucket in AWS S3. I want to host some static files there but if they're not all there, redirect to my main server. Seems simple but I've followed the instructions with no luck.

On the bucket I've given everyone view permissions.

I've added a bucket policy:

{
    "Version": "2012-10-17",
    "Id": "Policy999999999",
    "Statement": [
        {
            "Sid": "Stmt9999999999",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/*"
        }
    ]
}

On the Static Website Hosting, I've enabled it and created simple index.htm and error.htm pages (not that I need them but in case it's required). Then added these redirection rules:

<RoutingRules>
    <RoutingRule>
        <Condition>
            <HttpErrorCodeReturnedEquals>403</HttpErrorCodeReturnedEquals>
        </Condition>
        <Redirect>
            <Protocol>http</Protocol>
            <HostName>www.mydomain.com</HostName>
            <ReplaceKeyPrefixWith>/</ReplaceKeyPrefixWith>
        </Redirect>
    </RoutingRule>
</RoutingRules>

(apparently you need 403 not 404 if it doesn't have list permissions but it doesn't work on any variation of 403/404 and having list permissions on or not)

When I go to a URL on the bucket that doesn't exist (ones that exist are fine) I just get:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>9DD57C7132E76D65</RequestId>
<HostId>
I15odykUdMnGCC+hQfb1pfsnt+gkfGVxuD2sy5PrxrX16P8Njpy3yQiLLH30evTkdOOAl28HA9M=
</HostId>
</Error>

(or NoSuchKey if list permissions are on)

Help!

Check the console under web site hosting settings to find the web site endpoint hostname for your bucket.

You are sending requests to the REST endpoint hostname -- as evidenced by the XML error message.

http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteEndpoints.html

Also, you'll want to replace the key prefix with nothing, not a slash, or you'll end up with two slashes.