Unable to Find and Replace a string in 2GB XML fil

Question

I am new to Windows PowerShell. I am trying to perform a find and replace string on 4 occasions. But even a simple find and replace is throwing an

Exception of type 'System.OutOfMemoryException' was thrown.

error. I used Get-content.
Is there a way to achieve it without disrupting the memory?

e.g. Replace ".000000000Z" with ".732Z" where 732 will be the milliseconds the job is run?

PSversion: 3.0

Answer 1

The typical method is to use .Net methods to do it line by line.

Assuming you've got sensible line breaks, you can do something like this:

$FilePath = '...';
$NewFilePath = '...';

$Reader = New-Object -TypeName System.IO.StreamReader -ArgumentList $FilePath;
$Writer = New-Object -TypeName System.IO.StreamWriter -ArgumentList $NewFilePath;

while (($Line = $Reader.ReadLine()) -ne $null) {
    $Writer.WriteLine($Line.Replace('.000000000Z','.732Z'));
}

$Reader.Close();
$Writer.Close();

If your XML file is a single line of text, it gets more complicated.

Answer 2

Get-Content loads the entire file content into RAM to be acted upon.

You need to upgrade your RAM Use a deferent method and there are a few of them.

Get-Content
Get-Content -Raw
Get-Content -ReadCount
Switch -File

The .Net reader is the most optimal

[System.IO.File]::ReadAllText()
[System.IO.File]::ReadAllLines()
[System.IO.File]::ReadLines()
[System.IO.File]::OpenText().readtoend()

System.IO.File.ReadLines() is more than likely your best choice as it returns all the lines of a file, but lets you begin iterating over the lines immediately which means it does not have to store the entire contents in memory. More details here: https://msdn.microsoft.com/en-us/library/dd383503.aspx

Requires .NET 4.0 or higher.
foreach ($line in [System.IO.File]::ReadLines($filename)) {
    # do something with $line
}

So, you could do something like this...

$reader = [System.IO.File]::OpenText("my.log")
try {
    for() {
        $line = $reader.ReadLine()
        if ($line -eq $null) { break }
        # process the line
        $line
    }
}
finally {
    $reader.Close()
}

Or shorten it to this...

$reader = [System.IO.File]::OpenText("my.log")
while($null -ne ($line = $reader.ReadLine())) {
    $line
}