24GB的tar.gz解压使用sharpziplib(24GB tar.gz Decompress

2019-09-29 13:12发布

我有一个当前进程中提取tar.gz,而当前过程不能超过4GB处理......我想知道我在做什么错误导致我开始使用sharpzip一个错误的lib它说,参数长度不能少是零....所有的注释掉的代码是当前进程

请看下面的代码,给我方向

public static void ExtractTarGZFiles(string strExtractionPath, string strInboundFolder)
    {
        List<string> files = new List<string>();
        string strFile = Path.GetFullPath(ConfigurationManager.AppSettings["InboundFolderPath"].ToString());
        if (mblnRunForFile)
        {
            System.IO.FileInfo ioInfo = new FileInfo(mstrFilename);
            strFile = ioInfo.DirectoryName;
            files.Add(ioInfo.Name);

        }
        else
        {
            files = System.IO.Directory.GetFiles(ConfigurationManager.AppSettings["InboundFolderPath"], "*.gz").ToList();
        }

        foreach (string file in files)
        {
            try
            {
            //string stFilePath = file;
            //string testingthisdamnthing = strInboundFolder + mstrFilename;
            //FileStream xstream = new FileStream(stFilePath, FileMode.Open, FileAccess.Read, FileShare.None);
            //xstream.Close();
            //string strTemp = ConfigurationManager.AppSettings["ExtractTempFolderPath"];
            //TarArchive objTA = TarArchive.CreateInputTarArchive(new GZipStream(new FileStream(stFilePath, FileMode.Open, FileAccess.Read), CompressionMode.Decompress));
            //objTA.ProgressMessageEvent += ExtractTarNotifier;
             //objTA.ExtractContents(strTemp);



            //DirectoryInfo dirtemp = new DirectoryInfo(strTemp);
            DirectoryInfo dirExtract = new DirectoryInfo(strExtractionPath);

            Stream inStream = File.OpenRead(stFilePath);
            Stream gzipStream = new GZipInputStream(inStream);

            TarArchive tarArchive = TarArchive.CreateInputTarArchive(gzipStream);
            tarArchive.ExtractContents(ConfigurationManager.AppSettings["ExtractFolderPath"]);
            tarArchive.Close();

            gzipStream.Close();
            inStream.Close();
            //while (IsFileExistsinTempPath(dirtemp, dirExtract))
            //{
            //    //Do nothing
            //}
            //CopyFilesFromTempToExtract(dirtemp, dirExtract);
            //objTA.Close();
            //Logger.Write(" Tar.Gz files Decompressed Successfully");
            MonthLog.Log("Tar.Gz files Decompressed Successfully", "Month", 3, 2, System.Diagnostics.TraceEventType.Information, mstrFilename);
            System.IO.File.Copy(stFilePath, ConfigurationManager.AppSettings["ArchiveFolderPath"] + new FileInfo(stFilePath).Name, true);
            File.Delete(stFilePath);
            //Logger.Write(" Tar.GZ files Moved to Archive Folder");
            MonthlyGreenPackageLog.Log("Tar.Gz files Moved to Archive Folder", "Month", 3, 2, System.Diagnostics.TraceEventType.Information, mstrFilename);

            }
                catch (System.IO.IOException ex)
                {
                    //go to next file
                    //Logger.Write("Unable to open compressed file");
                    MonthLog.Log("Unable to open compressed file", "Month", 1, 1, System.Diagnostics.TraceEventType.Error, mstrFilename);
                    Email objEmail1 = new Email();
                    objEmail1.IsBodyHTML = true;
                    objEmail1.FromAddress = ConfigurationManager.AppSettings["FromAddress"];
                    string[] strToAddresses = ConfigurationManager.AppSettings["ExceptionAddress"].Split(',');
                    objEmail1.SetToAddress(strToAddresses);
                    objEmail1.Subject = "The Month  File  " + mstrFilename + " Failed to Decompress ";
                    objEmail1.Body = " Exception " + ex.Message + " occured while decompressing file";
                    //objEmail.AddAttachment("Exception occured while processingfiles");
                    objEmail1.SendEmail();
                    //Logger.Write("Sent a mail to all the Address");
                    throw ex;
                }
                catch (Exception ex)
                {
                    //Logger.Write("Exception " + ex.Message + " occured while decompressing file");
                    MonthlyGreenPackageLog.Log("Exception " + ex.Message + " occured while decompressing file", "Monthl", 1, 1, System.Diagnostics.TraceEventType.Error, mstrFilename);
                    Email objEmail = new Email();
                    objEmail.IsBodyHTML = true;
                    objEmail.FromAddress = ConfigurationManager.AppSettings["FromAddress"];
                    string[] strToAddresses = ConfigurationManager.AppSettings["ExceptionAddress"].Split(',');
                    objEmail.SetToAddress(strToAddresses);
                    objEmail.Subject = "The tar.gz Month File  " + mstrFilename + " Failed to Decompress ";
                    objEmail.Body = " Exception " + ex.Message + " occured while decompressing file";
                    //objEmail.AddAttachment("Exception occured while processing ADX files");
                    objEmail.SendEmail();
                    //Logger.Write("Sent a mail to all the Address");
                    throw ex;
                }

            }
        }

Answer 1:

我对你有三个潜在的解决方案,但我不能告诉你是否有任何人会实际工作。

  1. 使用从流解压例如,通过内存流,所以你永远不需要加载整个事情(这就是为什么现在失败)。
  2. 切换到使用DotNetZip ,这可能有处理x86计算机上的大文件更多的选择。
  3. 找哪家可以编写脚本或可监控的目录,并将其解压它得到您的应用程序之前一个普通unzipper应用。


Answer 2:

如果您的应用程序的x86,那么你只能有每个进程的4GB内存。 编译为64位,以使用更多的资源。



Answer 3:

这是从你使用的ZipLibrary。 您需要将应用程序编译为64位程序在64位环境中运行它。

    // To permit the zip to be unpacked by built-in extractor in WinXP and Server2003, WinZip 8, Java, and other older code,
    // you need to do one of the following: Specify UseZip64.Off, or set the Size.
    // If the file may be bigger than 4GB, or you do not need WinXP built-in compatibility, you do not need either,
    // but the zip will be in Zip64 format which not all utilities can understand.


文章来源: 24GB tar.gz Decompress using sharpziplib