Powershell read file in chunks

2019-05-31 19:48发布

问题:

I've had a script written in Powershell which transferred a file via FTP which worked fine by using:

$content = [System.IO.File]::ReadAllBytes($backup_app_data)

But this stopped working once the file size reached 2Gb, by throwing an error saying that this method is limited to reading files up to this size.

Exception calling "ReadAllBytes" with "1" argument(s): "The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

Now, I'm trying to modify my script and use an alternate approach with Read, which from what I understand will allow me to read the file in chunks and then write those chunks and so on.

Unfortunately although the script I've modified seems to be working, since a file is created on my FTP location, there is no data being transferred and no error is thrown by the script.

There is just a 0kb file created in the destination folder and the script ends.

I've tried to do some debugging, but I can't seem to find the issue. Below is the code I'm currently using.

$file = 'G:\Backups\DATA_backup_2016_09_05.zip'
$dest_name = 'DATA_backup_2016_09_05.zip'

$ftp = [System.Net.FtpWebRequest]::Create($ftp+$dest_name)
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.UseBinary = $true
$ftp.UsePassive = $true

# determine the size of the file
$file_size = (Get-Item $file).length
$chunk_size = 512mb

$bytes_to_read = $file_size
$iterations = $file_size / $chunk_size
$iter = 0

$fstream = [System.IO.FileStream]
[byte[]] $byte_array 

while ($bytes_to_read > 0){
    if($iterations > 1) {
        $content = $fstream.Read($byte_array, $iter, $chunk_size)
        }
    else {
        $content = $fstream.Read($byte_array, $iter, $bytes_to_read)
    }

    $ftp.ContentLength = $content.Length

    $rs = $ftp.GetRequestStream()
    $rs.Write($content, 0, $content.Length)

    # keep the loop going
    $iter = $iter + 1
    $iterations = $iterations - 1
    $bytes_to_read = $bytes_to_read - $chunk_size
}

$rs.Close()
$rs.Dispose()

Any help is appreciated.

回答1:

So, based on the suggestions in the comment section and using as an example the answer from the possible duplicate question, I managed to find the solution myself.

Below is the code I used to transfer via FTP a .zip archive which was 3.74Gb.

$ftp_addr = "ftp://ftp.somerandomwebsite.com/folder1/"
$user = "usr"
$pass = "passwrd"
$bufSize = 256mb

# ... missing code where I identify the file I want to FTP ... #

# Initialize connection to FTP
$ftp = [System.Net.FtpWebRequest]::Create($ftp_addr + $backup_file + ".zip")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.Timeout = -1              #infinite timeout
$ftp.ReadWriteTimeout = -1     #infinite timeout

$ftp.UseBinary = $true
$ftp.UsePassive = $true

$requestStream = $ftp.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file_to_ftp)
$chunk = New-Object byte[] $bufSize

while ( $bytesRead = $fileStream.Read($chunk, 0, $bufsize) ){
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
}

$FileStream.Close()
$requestStream.Close()

So far, this code has worked flawlessly for about multiple (20+) attempts. I hope it helps others!