Save all files in Visual Studio project as UTF-8

Question

I wonder if it's possible to save all files in a Visual Studio 2008 project into a specific character encoding. I got a solution with mixed encodings and I want to make them all the same (UTF-8 with signature).

I know how to save single files, but how about all files in a project?

Answer 1

Using C#:
1) Create a new ConsoleApplication, then install Mozilla Universal Charset Detector
2) Run code:

static void Main(string[] args)
{
    const string targetEncoding = "utf-8";
    foreach (var f in new DirectoryInfo(@"<your project's path>").GetFiles("*.cs", SearchOption.AllDirectories))
    {
        var fileEnc = GetEncoding(f.FullName);
        if (fileEnc != null && !string.Equals(fileEnc, targetEncoding, StringComparison.OrdinalIgnoreCase))
        {
            var str = File.ReadAllText(f.FullName, Encoding.GetEncoding(fileEnc));
            File.WriteAllText(f.FullName, str, Encoding.GetEncoding(targetEncoding));
        }
    }
    Console.WriteLine("Done.");
    Console.ReadKey();
}

private static string GetEncoding(string filename)
{
    using (var fs = File.OpenRead(filename))
    {
        var cdet = new Ude.CharsetDetector();
        cdet.Feed(fs);
        cdet.DataEnd();
        if (cdet.Charset != null)
            Console.WriteLine("Charset: {0}, confidence: {1} : " + filename, cdet.Charset, cdet.Confidence);
        else
            Console.WriteLine("Detection failed: " + filename);
        return cdet.Charset;
    }
}

Answer 2

I would convert the files programmatically (outside VS), e.g. using a Python script:

import glob, codecs

for f in glob.glob("*.py"):
    data = open("f", "rb").read()
    if data.startswith(codecs.BOM_UTF8):
        # Already UTF-8
        continue
    # else assume ANSI code page
    data = data.decode("mbcs")
    data = codecs.BOM_UTF8 + data.encode("utf-8")
    open("f", "wb").write(data)

This assumes all files not in "UTF-8 with signature" are in the ANSI code page - this is the same what VS 2008 apparently also assumes. If you know that some files have yet different encodings, you would have to specify what these encodings are.

Answer 3

adapted the version above to make it work.

// important! create a utf8 encoding that explicitly writes no BOM            
var utf8nobom = new UTF8Encoding(false); 
foreach (var f in new DirectoryInfo(dir).GetFiles("*.*", SearchOption.AllDirectories))
{
    string text = File.ReadAllText(f.FullName);
    File.WriteAllText(f.FullName, text, utf8nobom);
}

Answer 4

I'm only offering this suggestion in case there's no way to automatically do this in Visual Studio (I'm not even sure this would work):

1. Create a class in your project named 足の不自由なハッキング (or some other unicode text that will force Visual Studio to encode as UTF-8).
2. Add "using MyProject.足の不自由なハッキング;" to the top of each file. You should be able to do it on everything by doing a global replace of "using System.Text;" with "using System.Text;using MyProject.足の不自由なハッキング;".
3. Save everything. You may get a long string of "Do you want to save X.cs using UTF-8?" messages or something.

Answer 5

Experienced encoding problems after converting solution from VS2008 to VS2015. After conversion all project files was encoded in ANSI, but they contained UTF8 content and was recongnized as ANSI files in VS2015. Tried many conversion tactics, but worked only this solution.

 Encoding encoding = Encoding.Default;
 String original = String.Empty;
 foreach (var f in new DirectoryInfo(path).GetFiles("*.cs", SearchOption.AllDirectories))
 {
    using (StreamReader sr = new StreamReader(f.FullName, Encoding.Default))
    {
       original = sr.ReadToEnd();
       encoding = sr.CurrentEncoding;
       sr.Close();
    }
    if (encoding == Encoding.UTF8)
       continue;
    byte[] encBytes = encoding.GetBytes(original);
    byte[] utf8Bytes = Encoding.Convert(encoding, Encoding.UTF8, encBytes);
    var utf8Text = Encoding.UTF8.GetString(utf8Bytes);

    File.WriteAllText(f.FullName, utf8Text, Encoding.UTF8);
 }

Answer 6

In case you need to do this in PowerShell, here is my little move:

Function Write-Utf8([string] $path, [string] $filter='*.*')
{
    [IO.SearchOption] $option = [IO.SearchOption]::AllDirectories;
    [String[]] $files = [IO.Directory]::GetFiles((Get-Item $path).FullName, $filter, $option);
    foreach($file in $files)
    {
        "Writing $file...";
        [String]$s = [IO.File]::ReadAllText($file);
        [IO.File]::WriteAllText($file, $s, [Text.Encoding]::UTF8);
    }
}