Appending files to a zip file with Java

Question

I am currently extracting the contents of a war file and then adding some new files to the directory structure and then creating a new war file.

This is all done programatically from Java - but I am wondering if it wouldn't be more efficient to copy the war file and then just append the files - then I wouldn't have to wait so long as the war expands and then has to be compressed again.

I can't seem to find a way to do this in the documentation though or any online examples.

Anyone can give some tips or pointers?

UPDATE:

TrueZip as mentioned in one of the answers seems to be a very good java library to append to a zip file (despite other answers that say it is not possible to do this).

Anyone have experience or feedback on TrueZip or can recommend other similar libaries?

Answer 1

In Java 7 we got Zip File System that allows adding and changing files in zip (jar, war) without manual repackaging.

We can directly write to files inside zip files as in the following example.

Map<String, String> env = new HashMap<>(); 
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env))
{
    Path nf = fs.getPath("new.txt");
    try (Writer writer = Files.newBufferedWriter(nf, StandardCharsets.UTF_8, StandardOpenOption.CREATE)) {
        writer.write("hello");
    }
}

Answer 2

I don't know of a Java library that does what you describe. But what you described is practical. You can do it in .NET, using DotNetZip.

Michael Krauklis is correct that you cannot simply "append" data to a war file or zip file, but it is not because there is an "end of file" indication, strictly speaking, in a war file. It is because the war (zip) format includes a directory, which is normally present at the end of the file, that contains metadata for the various entries in the war file. Naively appending to a war file results in no update to the directory, and so you just have a war file with junk appended to it.

What's necessary is an intelligent class that understands the format, and can read+update a war file or zip file, including the directory as appropriate. DotNetZip does this, without uncompressing/recompressing the unchanged entries, just as you described or desired.

Answer 3

Here is Java 1.7 version of Liam answer which uses try with resources and Apache Commons IO.

The output is written to a new zip file but it can be easily modified to write to the original file.

  /**
   * Modifies, adds or deletes file(s) from a existing zip file.
   *
   * @param zipFile the original zip file
   * @param newZipFile the destination zip file
   * @param filesToAddOrOverwrite the names of the files to add or modify from the original file
   * @param filesToAddOrOverwriteInputStreams the input streams containing the content of the files
   * to add or modify from the original file
   * @param filesToDelete the names of the files to delete from the original file
   * @throws IOException if the new file could not be written
   */
  public static void modifyZipFile(File zipFile,
      File newZipFile,
      String[] filesToAddOrOverwrite,
      InputStream[] filesToAddOrOverwriteInputStreams,
      String[] filesToDelete) throws IOException {


    try (ZipOutputStream out = new ZipOutputStream(new FileOutputStream(newZipFile))) {

      // add existing ZIP entry to output stream
      try (ZipInputStream zin = new ZipInputStream(new FileInputStream(zipFile))) {
        ZipEntry entry = null;
        while ((entry = zin.getNextEntry()) != null) {
          String name = entry.getName();

          // check if the file should be deleted
          if (filesToDelete != null) {
            boolean ignoreFile = false;
            for (String fileToDelete : filesToDelete) {
              if (name.equalsIgnoreCase(fileToDelete)) {
                ignoreFile = true;
                break;
              }
            }
            if (ignoreFile) {
              continue;
            }
          }

          // check if the file should be kept as it is
          boolean keepFileUnchanged = true;
          if (filesToAddOrOverwrite != null) {
            for (String fileToAddOrOverwrite : filesToAddOrOverwrite) {
              if (name.equalsIgnoreCase(fileToAddOrOverwrite)) {
                keepFileUnchanged = false;
              }
            }
          }

          if (keepFileUnchanged) {
            // copy the file as it is
            out.putNextEntry(new ZipEntry(name));
            IOUtils.copy(zin, out);
          }
        }
      }

      // add the modified or added files to the zip file
      if (filesToAddOrOverwrite != null) {
        for (int i = 0; i < filesToAddOrOverwrite.length; i++) {
          String fileToAddOrOverwrite = filesToAddOrOverwrite[i];
          try (InputStream in = filesToAddOrOverwriteInputStreams[i]) {
            out.putNextEntry(new ZipEntry(fileToAddOrOverwrite));
            IOUtils.copy(in, out);
            out.closeEntry();
          }
        }
      }

    }

  }

Answer 4

As others mentioned, it's not possible to append content to an existing zip (or war). However, it's possible to create a new zip on the fly without temporarily writing extracted content to disk. It's hard to guess how much faster this will be, but it's the fastest you can get (at least as far as I know) with standard Java. As mentioned by Carlos Tasada, SevenZipJBindings might squeeze out you some extra seconds, but porting this approach to SevenZipJBindings will still be faster than using temporary files with the same library.

Here's some code that writes the contents of an existing zip (war.zip) and appends an extra file (answer.txt) to a new zip (append.zip). All it takes is Java 5 or later, no extra libraries needed.

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;

public class Main {

    // 4MB buffer
    private static final byte[] BUFFER = new byte[4096 * 1024];

    /**
     * copy input to output stream - available in several StreamUtils or Streams classes 
     */    
    public static void copy(InputStream input, OutputStream output) throws IOException {
        int bytesRead;
        while ((bytesRead = input.read(BUFFER))!= -1) {
            output.write(BUFFER, 0, bytesRead);
        }
    }

    public static void main(String[] args) throws Exception {
        // read war.zip and write to append.zip
        ZipFile war = new ZipFile("war.zip");
        ZipOutputStream append = new ZipOutputStream(new FileOutputStream("append.zip"));

        // first, copy contents from existing war
        Enumeration<? extends ZipEntry> entries = war.entries();
        while (entries.hasMoreElements()) {
            ZipEntry e = entries.nextElement();
            System.out.println("copy: " + e.getName());
            append.putNextEntry(e);
            if (!e.isDirectory()) {
                copy(war.getInputStream(e), append);
            }
            append.closeEntry();
        }

        // now append some extra content
        ZipEntry e = new ZipEntry("answer.txt");
        System.out.println("append: " + e.getName());
        append.putNextEntry(e);
        append.write("42\n".getBytes());
        append.closeEntry();

        // close
        war.close();
        append.close();
    }
}

Answer 5

As Cheeso says, there's no way of doing it. AFAIK the zip front-ends are doing exactly the same as you internally.

Anyway if you're worried about the speed of extracting/compressing everything, you may want to try the SevenZipJBindings library.

I covered this library in my blog some months ago (sorry for the auto-promotion). Just as an example, extracting a 104MB zip file using the java.util.zip took me 12 seconds, while using this library took 4 seconds.

In both links you can find examples about how to use it.

Hope it helps.

Answer 6

See this bug report.

Using append mode on any kind of structured data like zip files or tar files is not something you can really expect to work. These file formats have an intrinsic "end of file" indication built into the data format.

If you really want to skip the intermediate step of un-waring/re-waring, you could read the war file file, get all the zip entries, then write to a new war file "appending" the new entries you wanted to add. Not perfect, but at least a more automated solution.