Create a large zip assembly with maven-assembly-pl

Question

I'm having a problem with creating a larger zip assembly (uncompressed takes over 3GB) using maven-assembly-plugin. The problem occurs when building the output zip file (compressed less than 1GB). Running maven with option -e gives me a more detailed info:

[INFO] Building zip: xxx/HG19-UCSC-dist.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:40.199s
[INFO] Finished at: Wed Oct 02 11:08:44 BST 2013
[INFO] Final Memory: 13M/723M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (make-assembly) on project HG19: Execution make-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single failed: invalid entry size -> [Help 1]
...
...
...
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution make-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single failed: invalid entry size
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:115)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
    ... 19 more
Caused by: java.lang.IllegalArgumentException: invalid entry size
    at java.util.zip.ZipEntry.setSize(ZipEntry.java:135)
    at org.codehaus.plexus.archiver.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:352)
    at org.codehaus.plexus.archiver.zip.ZipOutputStream.finish(ZipOutputStream.java:316)

Looking around I found that the issue comes from missing support for Zip64 in my JDK/JRE (https://blogs.oracle.com/xuemingshen/entry/zip64_support_for_4g_zipfile), which was added to OpenJDK in 2011.

However, I've updated my JDK to the newest one provided by Oracle:

$ mvn -version
Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28 03:15:32+0100)
Maven home: /usr/local/apache-maven-3.1.0
Java version: 1.7.0_40, vendor: Oracle Corporation
Java home: /usr/local/java/jdk1.7.0_40/jre
Default locale: en_GB, platform encoding: ISO-8859-1
OS name: "linux", version: "2.6.32-279.2.1.el6.x86_64", arch: "amd64", family: "unix"

and it reports the same issue, again. Does the newest Oracle JDK have no support for Zip64 or it's something else?

I guess, I could try building OpenJDK b147 from sources but would like to avoid that unless inevitable.

Answer 1

I found the culprit, so reporting the issue just to save others time.

It seems that the problem lays in org.codehaus.plexus.archiver.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:352) as reported in my exception stack trace. Looking at the sources of org.codehaus...ZipOutputStream.closeEntry I can see:

entry.setSize( def.getTotalIn() );

and that def is java.util.zip.Deflater. However, Deflater.getTotalIn returns int not long. Instead the code should use Deflater.getBytesRead() which I'm gonna report to the developers.

[EDIT] It's been a while but I see people reading this, so FYI the problem has been fixed since version 2.5 of the maven assembly plugin.

Answer 2

I had a similar issue (same error in the OP) when using the shade plugin that was caused by not running mvn clean before packaging.

Answer 3

As we can see here, (ZipEntry class), this has been fixed in the JDK 1.7.

You've shown that your mvn run from command line shows the JDK version 1.7.

Are you trying to complete build in the same manner, I mean from command line (and not IDE, which can use another JDK for maven?)

Answer 4

Encountered the same issue. A very big (3GB) log file was occasionally picked up when building the project's jar, and this caused the issue. No error after the file had been deleted and the size of the target jar is back to under 10Mb.