Multipart File Upload on Google Appengine using je

Question

I wrote an application on Google Appengine with Jersey to handle simple file uploading. This works fine when it was on jersey 1.2. In the later versions (current 1.7) @FormDataParam is introduced to handle multipart/form inputs. I am using jersey-multipart and the mimepull dependency. It seems that the new way of doing it is creating temporary files in appengine which we all know is illegal...

Am I missing something or doing something wrong here since Jersey is now supposedly compatible with AppEngine?

@POST 
@Path("upload") 
@Consumes(MediaType.MULTIPART_FORM_DATA) 
public void upload(@FormDataParam("file") InputStream in) { .... }

The above will fail when called with these exceptions...

/upload
java.lang.SecurityException: Unable to create temporary file
    at java.io.File.checkAndCreate(File.java:1778)
    at java.io.File.createTempFile(File.java:1870)
    at java.io.File.createTempFile(File.java:1907)
    at org.jvnet.mimepull.MemoryData.createNext(MemoryData.java:87)
    at org.jvnet.mimepull.Chunk.createNext(Chunk.java:59)
    at org.jvnet.mimepull.DataHead.addBody(DataHead.java:82)
    at org.jvnet.mimepull.MIMEPart.addBody(MIMEPart.java:192)
    at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:235)
    at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:176)
    at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:101)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readMultiPart(MultiPartReaderClientSide.java:177)
    at com.sun.jersey.multipart.impl.MultiPartReaderServerSide.readMultiPart(MultiPartReaderServerSide.java:80)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:139)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:77)
    at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
    at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:538)

Anyone have a clue? Is there a way to do thing while preventing mimepull from creating the temporary file?

Answer 1

i've found solution to programmatically avoid to use temporary file creation (very useful for GAE implementation)

My solution consist of creating a new MultiPartReader Provider ... below my code

---

  @Provider
    @Consumes("multipart/*")
    public class GaeMultiPartReader implements MessageBodyReader<MultiPart> {

    final Log logger = org.apache.commons.logging.LogFactory.getLog(getClass());

    private final Providers providers;

    private final CloseableService closeableService;

    private final MIMEConfig mimeConfig;

    private String getFixedHeaderValue(Header h) {
        String result = h.getValue();

        if (h.getName().equals("Content-Disposition") && (result.indexOf("filename=") != -1)) {
            try {
                result = new String(result.getBytes(), "utf8");
            } catch (UnsupportedEncodingException e) {            
                final String msg = "Can't convert header \"Content-Disposition\" to UTF8 format.";
                logger.error(msg,e);
                throw new RuntimeException(msg);
            }
        }

        return result;
    }

    public GaeMultiPartReader(@Context Providers providers, @Context MultiPartConfig config,
        @Context CloseableService closeableService) {
        this.providers = providers;

        if (config == null) {
            final String msg = "The MultiPartConfig instance we expected is not present. "
                + "Have you registered the MultiPartConfigProvider class?";
            logger.error( msg );
            throw new IllegalArgumentException(msg);
        }
        this.closeableService = closeableService;

        mimeConfig = new MIMEConfig();
        //mimeConfig.setMemoryThreshold(config.getBufferThreshold());
        mimeConfig.setMemoryThreshold(-1L); // GAE FIX
    }

    @Override
    public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return MultiPart.class.isAssignableFrom(type);
    }

    @Override
    public MultiPart readFrom(Class<MultiPart> type, Type genericType, Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, String> headers, InputStream stream) throws IOException, WebApplicationException {
        try {
            MIMEMessage mm = new MIMEMessage(stream, mediaType.getParameters().get("boundary"), mimeConfig);

            boolean formData = false;
            MultiPart multiPart = null;

            if (MediaTypes.typeEquals(mediaType, MediaType.MULTIPART_FORM_DATA_TYPE)) {
                multiPart = new FormDataMultiPart();
                formData = true;
            } else {
                multiPart = new MultiPart();
            }

            multiPart.setProviders(providers);

            if (!formData) {
                multiPart.setMediaType(mediaType);
            }

            for (MIMEPart mp : mm.getAttachments()) {
                BodyPart bodyPart = null;

                if (formData) {
                    bodyPart = new FormDataBodyPart();
                } else {
                    bodyPart = new BodyPart();
                }

                bodyPart.setProviders(providers);

                for (Header h : mp.getAllHeaders()) {
                    bodyPart.getHeaders().add(h.getName(), getFixedHeaderValue(h));
                }

                try {
                    String contentType = bodyPart.getHeaders().getFirst("Content-Type");

                    if (contentType != null) {
                        bodyPart.setMediaType(MediaType.valueOf(contentType));
                    }

                    bodyPart.getContentDisposition();
                } catch (IllegalArgumentException ex) {
                    logger.error( "readFrom error", ex );
                    throw new WebApplicationException(ex, 400);
                }

                bodyPart.setEntity(new BodyPartEntity(mp));
                multiPart.getBodyParts().add(bodyPart);
            }

            if (closeableService != null) {
                closeableService.add(multiPart);
            }

            return multiPart;
        } catch (MIMEParsingException ex) {
            logger.error( "readFrom error", ex );
            throw new WebApplicationException(ex, 400);
        }
    }

}

Answer 2

It is very important to put the file jersey-multipart-config.properties under WEB-INF/classes inside the WAR.

Usually in a WAR file structure you put the config files (web.xml, appengine-web.xml) into WEB-INF/, but here you need to put into WEB-INF/classes.

Example Maven configuration:

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-war-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <archiveClasses>true</archiveClasses>
                <webResources>
                    <resource>
                        <directory>${basedir}/src/main/webapp/WEB-INF</directory>
                        <filtering>true</filtering>
                        <targetPath>WEB-INF</targetPath>
                    </resource>
                    <resource>
                        <directory>${basedir}/src/main/resources</directory>
                        <targetPath>WEB-INF/classes</targetPath>
                    </resource>
                </webResources>
            </configuration>
        </plugin>

And your project structure can look like:

Content of jersey-multipart-config.properties with Jersey 2.x:

jersey.config.multipart.bufferThreshold = -1

Answer 3

For the benefit of those struggling when using Eclipse with GPE (Google Plugin for Eclipse) I give this slightly modified solution derived from @yves' answer.

I have tested it with App Engine SDK 1.9.10 and Jersey 2.12. It will not work with App Engine SDK 1.9.6 -> 1.9.9 amongst others due to a different issue.

Under your \war\WEB-INF\classes folder create a new file called jersey-multipart-config.properties. Edit the file so it contains the line jersey.config.multipart.bufferThreshold = -1.

Note that the \classes folder is hidden in Eclipse so look for the folder in your operating system's file explorer (e.g. Windows Explorer).

Now, both when the multipart feature gets initialized (on Jersey servlet initialization) and when a file upload is done (on Jersey servlet post request) the temp file will not be created anymore and GAE won't complain.

Answer 4

We experienced a similar problem, Jetty wouldn't let us upload files more than 9194 bytes, (all of a sudden - one day), we realised afterwards that someone had taken our user access from /tmp, which corresponds to java.io.tmpdir on some linux versions, so Jetty couldn't store the uploaded file there, and we got a 400 error.

Answer 5

For files beyond its default size, multipart will create a temporary file. To avoid this — creating a file is impossible on gae — you can create a jersey-multipart-config.properties file in the project's resources folder and add this line to it:

bufferThreshold = -1

Then, the code is the one you gave:

@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
public Response post(@FormDataParam("file") InputStream stream, @FormDataParam("file") FormDataContentDisposition disposition) throws IOException {
  post(file, disposition.getFileName());
  return Response.ok().build();
}