Percent-encoded slash (“/”) is decoded before the

Question

I have an URL containing several slash characters (/) as a part of the filename (not the URL). But when I send http request, the percent-encoded %2F is translated to / before the request dispatch, therefore generating a wrong URL.

How can I make a literal http request ignoring the percent-encoded values in PowerShell?

Actual URL used (Chromium browser):

https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media

---

I have tried Invoke-WebRequest cmdlet:

Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64/292817/chrome-win32.zip?generation=1409504089694000&alt=media with 0-byte payload1`

Not found error.

Also tried WebClient's DownloadFile method:

$wclient = New-Object System.Net.WebClient
$wclient.DownloadFile($ChromeUrl, $FilePath)

Returns 404 due to wrong URL requested again.

---

Workaround 1 (successful)

Reflection-based workarounds provided by briantist and Tanuj Mathur are both working great. The latter one:

$UrlFixSrc = @" 
using System;
using System.Reflection;

public static class URLFix 
{ 
    public static void ForceCanonicalPathAndQuery(Uri uri)
    {
        string paq = uri.PathAndQuery;
        FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
        ulong flags = (ulong) flagsFieldInfo.GetValue(uri);
        flags &= ~((ulong) 0x30);
        flagsFieldInfo.SetValue(uri, flags);
    }
} 
"@ 

Add-Type -TypeDefinition $UrlFixSrc-Language CSharp
[URLFix]::ForceCanonicalPathAndQuery([URI]$ChromeUrl)

Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292640%2Fchrome-win32.zip?generation=1409351584147000&alt=media

Workaround 2 (successful)

More clean solution (offered by Tanuj Mathur), but requires access to system files, is by adding a config file %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe.config with the following content:

<?xml version="1.0" encoding="utf-8" ?> 
 <configuration> 
   <uri>
     <schemeSettings>
      <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
      <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
     </schemeSettings>
   </uri>
</configuration>

Corresponding modifications has to be done in powerhsell_ise.exe.config for it to work in ISE.

Workaround 3 (failed)

I thought its a System.URI class constructor problem that is called upon implicit casting, which translates the escaped values. Tried an overloaded variant Uri ([String]uriString, [Boolean]dontEscape). But there was no difference. The same outcome with or without dontEscape argument.

$uri = new-object System.Uri($ChromeUrl, $true)
$uri | Format-List OriginalString, AbsoluteUri

  OriginalString : https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media
  AbsoluteUri    : https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64/292817/chrome-win32.zip?generation=1409504089694000&alt=media

---

Workaround 4 (failed)

Also tried to fool URI parser by replacing percent character with its percent-encoded value %25. But then it ignored everything completely.

Invoke-WebRequest -Uri $ChromeUrl.Replace('%', '%25') -OutFile $DownloadPath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%252F292817%252Fchrome-win32.zip?generation=1409504089694000&alt=media with 0-byte pa yload

---

Workaround 5 (not implemented)

The only way I found that requests URL properly is through Internet Explorer instance.

$ie = New-Object -ComObject InternetExplorer.Application                
$ie.Visible = $true
$ie.Silent = $false
$ie.Navigate2($ChromeUrl)

But then I don't know how to automate the 'Saves as' button click and save it to desired path. Also, even if implemented, I don't feel like this is a good solution. What happens when IE is already running or uninstalled from the system?

Answer 1

If you're going to use PowerShell you can also do Workaround 1 in pure PowerShell:

function UrlFix([Uri]$url) {
    $url.PathAndQuery | Out-Null
    $m_Flags = [Uri].GetField("m_Flags", $([Reflection.BindingFlags]::Instance -bor [Reflection.BindingFlags]::NonPublic))
    [uint64]$flags = $m_Flags.GetValue($url)
    $m_Flags.SetValue($url, $($flags -bxor 0x30))
}

UrlFix $ChromeUrl
Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose

Answer 2

Wow, this is quite a conundrum. There is a bug report about this on Microsoft Connect. It seems there is a workaround for ASP.net which won't help you in PowerShell.

Here's where it gets really weird though. I am running PowerShell 4.0. I can reproduce this problem when running in the console host. However, if I run the exact same code in the ISE Host, it works flawlessly.

I don't have any idea how or why. I even remoted into another system not on my network to make sure I didn't somehow change anything weird on my system. Same result. Bottom line:

$a = 'https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media'
Invoke-WebRequest -Uri $a

This works in ISE, does not work in console host. I even tried it with -UseBasicParsing to make sure it wasn't a weird quirk of DOM parsing.

A Dirty Workaround

I took the C# code in Simon Maurier's answer to "How to make System.Uri not to unescape %2f (slash) in path?" and I adapted it for use in PowerShell:

$uriFixerDef = @'
using System;
using System.Reflection;

public class UriFixer
{
    private const int UnEscapeDotsAndSlashes = 0x2000000;
    private const int SimpleUserSyntax = 0x20000;

    public static void LeaveDotsAndSlashesEscaped(Uri uri)
    {
        if (uri == null)
            throw new ArgumentNullException("uri");

        FieldInfo fieldInfo = uri.GetType().GetField("m_Syntax", BindingFlags.Instance | BindingFlags.NonPublic);
        if (fieldInfo == null)
            throw new MissingFieldException("'m_Syntax' field not found");

        object uriParser = fieldInfo.GetValue(uri);
        fieldInfo = typeof(UriParser).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
        if (fieldInfo == null)
            throw new MissingFieldException("'m_Flags' field not found");

        object uriSyntaxFlags = fieldInfo.GetValue(uriParser);

        // Clear the flag that we do not want
        uriSyntaxFlags = (int)uriSyntaxFlags & ~UnEscapeDotsAndSlashes;
        uriSyntaxFlags = (int)uriSyntaxFlags & ~SimpleUserSyntax;
        fieldInfo.SetValue(uriParser, uriSyntaxFlags);
    }
}
'@

Add-Type -TypeDefinition $uriFixerDef

$u = 'https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media'

[UriFixer]::LeaveDotsAndSlashesEscaped($u)

Invoke-WebRequest -Uri $u

I first tested it in ISE and then later found that ISE was working no matter what. So I did in fact try this in a clean console host environment and before invoking the method, I got notfound. After invoking, worked.

As it says in the linked answer, this is an ugly hack, may break in future versions, etc.

I do hope it helps, this is an interesting issue.

Answer 3

I've been playing around with your code for the last few hours, and it's a doozy. The given code and it's variants all pass when run in the Powershell ISE, but fail on the Powershell console. The issue itself seems to be the one documented on Microsoft Connect here.

Interestingly, as per user Glenn Block's answer on a related issue, this bug was fixed in .NET Framework 4.5. You can check the version of the .NET framework being used by your Powershell by running the command $PSVersionTable. As long as the CLRVersion value is of the form 4.0.30319.x, where x > 1700, then you are running v4.5 of the framework.

I'm running Powershell v4.0 on .NET framework 4.5 on my machine, so that explains why Powershell ISE shows the correct behaviour, but I was not able to figure out why Powershell console does not. I verified the .NET assemblies loaded by both, and they seem to be the same.

As things stand, we have two options. One is to use reflection and set a private field on the .Net class to prevent this behaviour (as outlined in this answer). The other is to use the workaround listed in the Microsoft Connect issue. This involves the following steps:

1. Go to your Powershell install folder (this was "C:\Windows\System32\WindowsPowerShell\v1.0\" on my machine). This folder should have the file powershell.exe in it.
2. Create a new text file in this folder, and name it powershell.exe.config

3. Open this file in a text editor, and paste the following text into it: <?xml version="1.0" encoding="utf-8" ?> <configuration> <uri> <schemeSettings> <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" /> <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes" /> </schemeSettings> </uri> </configuration>

4. Save this file. Close ALL running instances of Powershell.

5. Start a new instance of Powershell. This will cause Powershell to detect the config file you created and parse it. The config entries basically tell the .NET libraries to disable the automatic unescaping of HTTP and HTTPS uri's.
6. Run your script. You should no longer see the issue with the Uris.