Resteasy multipart/data-form file upload on GAE

Question

I'm trying to use resteasy 2.0.1.GA to upload a form with a file in it into GAE application, using the method advised at How do I do a multipart/form file upload with jax-rs?

Index.html

<form action="/rest/upload" method="post" enctype="multipart/form-data">
  <input type="text" name="name" />
  <input type="file" name="file" />
  <input type="submit" />
</form>

Rest.java

@Path("")
public class Rest {
    @POST
    @Path("/rest/upload")
    @Consumes("multipart/form-data")
    public String postContent(@MultipartForm UploadForm form) {
        System.out.println(form.getData().length);
        System.out.println(form.getName());
        return "Done";
    }
}

UploadForm.java

public class UploadForm {

    private String name;
    private byte[] data;

    @FormParam("name")
    public void setPath(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    @FormParam("file")
    public void setContentData(byte[] data) {
        this.data = data;
    }

    public byte[] getData() {
        return data;
    }
}

But I'm getting the following error message (probably due to the RESTEasy Provider's implmenetation that uses temporary files to handle the input stream):

HTTP ERROR 500

Problem accessing /files/service/upload. Reason:

    java.io.FileOutputStream is a restricted class. Please see the Google  App Engine developer's guide for more details.

Caused by:

java.lang.NoClassDefFoundError: java.io.FileOutputStream is a restricted class. Please see the Google  App Engine developer's guide for more details.
    at com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime.java:51)
    at org.apache.james.mime4j.storage.TempFileStorageProvider$TempFileStorageOutputStream.<init>(TempFileStorageProvider.java:117)
    at org.apache.james.mime4j.storage.TempFileStorageProvider.createStorageOutputStream(TempFileStorageProvider.java:107)
    at org.apache.james.mime4j.storage.ThresholdStorageProvider$ThresholdStorageOutputStream.write0(ThresholdStorageProvider.java:113)
    at org.apache.james.mime4j.storage.StorageOutputStream.write(StorageOutputStream.java:119)
    at org.apache.james.mime4j.codec.CodecUtil.copy(CodecUtil.java:43)
    at org.apache.james.mime4j.storage.AbstractStorageProvider.store(AbstractStorageProvider.java:57)
    at org.apache.james.mime4j.message.BodyFactory.textBody(BodyFactory.java:167)
    at org.apache.james.mime4j.message.MessageBuilder.body(MessageBuilder.java:148)
    at org.apache.james.mime4j.parser.MimeStreamParser.parse(MimeStreamParser.java:101)
    at org.apache.james.mime4j.message.Message.<init>(Message.java:141)
    at org.apache.james.mime4j.message.Message.<init>(Message.java:100)
    at org.jboss.resteasy.plugins.providers.multipart.MultipartInputImpl.parse(MultipartInputImpl.java:76)
    at org.jboss.resteasy.plugins.providers.multipart.MultipartFormAnnotationReader.readFrom(MultipartFormAnnotationReader.java:55)
    at org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl.proceed(MessageBodyReaderContextImpl.java:105)
    at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.read(GZIPDecodingInterceptor.java:46)
    at org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl.proceed(MessageBodyReaderContextImpl.java:108)
    at org.jboss.resteasy.core.messagebody.ReaderUtility.doRead(ReaderUtility.java:111)
    at org.jboss.resteasy.core.messagebody.ReaderUtility.doRead(ReaderUtility.java:93)
    at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:146)
    at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:114)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:252)
    at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:217)
    at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:206)
    at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:514)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:491)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:120)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:200)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:48)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:43)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    ...

Has anyone encountered this issue with GAE and RESTEasy? Has anyone solved it? I couldn't find any mentioning for this issue anywhere. Thanks!

Answer 1

It looks like the mime4j library is trying to write out temporary files, which is not allowed on app engine. mime4j can be configured to use a memory storage provider, but I don't know if the RESTeasy use of it allows that configuration.

Answer 2

I tried to use MemoryStorageProvider. But looks like it does not work for most files other than smaller ones.

I came up with another solution which is extending AbstractStorageProvider using google cloud storage and it works nicely.

https://gist.github.com/azimbabu/0aef75192c385c6d4461118583b6d22f

import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsInputChannel;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import lombok.extern.slf4j.Slf4j;
import org.apache.james.mime4j.storage.AbstractStorageProvider;
import org.apache.james.mime4j.storage.Storage;
import org.apache.james.mime4j.storage.StorageOutputStream;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.Channels;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import java.util.UUID;

/**
 * A {@link org.apache.james.mime4j.storage.StorageProvider} that stores the data in google cloud storage files. The files
 * are stored in a user specified bucket. User of this class needs to supply the google cloud storage service and bucket name.
 *
 * This implementation is based on {@link org.apache.james.mime4j.storage.TempFileStorageProvider}
 * <p>
 * Example usage:
 *
 * <pre>
 * final String bucketName = "my-bucket";
 * DefaultStorageProvider.setInstance(new GcsStorageProvider(gcsService, bucketName));
 * </pre>
 */
@Slf4j
public class GcsStorageProvider extends AbstractStorageProvider {

    private static final int FETCH_SIZE_MB = 4 * 1024 * 1024;

    private static final String PUBLIC_READ = "public-read";

    private static final GcsFileOptions gcsFileOpts = new GcsFileOptions.Builder().acl(PUBLIC_READ).mimeType("text/csv").build();

    private final GcsService gcsService;

    private final String bucketName;

    /**
     * Creates a new <code>GcsStorageProvider</code> using the given
     * values.
     *
     * @param gcsService an implementation of {@link GcsService}
     * @param bucketName google cloud storage bucket name to use.
     */
    public GcsStorageProvider(final GcsService gcsService, final String bucketName) {
        this.gcsService = gcsService;
        this.bucketName = bucketName;
    }

    @Override
    public StorageOutputStream createStorageOutputStream() throws IOException {
        return new GcsStorageProvider.GcsStorageOutputStream(gcsService, bucketName);
    }

    private static final class GcsStorage implements Storage {

        private final GcsService gcsService;

        private GcsFilename gcsFilename;

        private static final Set<GcsFilename> filesToDelete = new HashSet();

        public GcsStorage(final GcsService gcsService, final GcsFilename gcsFilename) {
            this.gcsService = gcsService;
            this.gcsFilename = gcsFilename;
        }

        @Override
        public InputStream getInputStream() throws IOException {
            if (this.gcsFilename == null) {
                throw new IllegalStateException("storage has been deleted");
            } else {
                final GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFilename, 0, FETCH_SIZE_MB);
                return Channels.newInputStream(readChannel);
            }
        }

        @Override
        public void delete() {
            synchronized(filesToDelete) {
                if (this.gcsFilename != null) {
                    filesToDelete.add(this.gcsFilename);
                    this.gcsFilename = null;
                }

                final Iterator iterator = filesToDelete.iterator();

                while(iterator.hasNext()) {
                    GcsFilename filename = (GcsFilename)iterator.next();
                    try {
                        if (gcsService.delete(filename)) {
                            iterator.remove();
                        }
                    } catch (final IOException ex) {
                        log.error(ex.getMessage(), ex);
                    }
                }

            }
        }
    }

    private static final class GcsStorageOutputStream extends StorageOutputStream {

        private final GcsService gcsService;

        private GcsFilename gcsFilename;

        private final OutputStream outputStream;

        public GcsStorageOutputStream(final GcsService gcsService, final String bucketName) throws IOException {
            this.gcsService = gcsService;

            final String fileName = UUID.randomUUID().toString();
            this.gcsFilename = new GcsFilename(bucketName, fileName);

            GcsOutputChannel gcsOutputChannel = gcsService.createOrReplace(gcsFilename, gcsFileOpts);
            this.outputStream = Channels.newOutputStream(gcsOutputChannel);
        }

        @Override
        protected void write0(byte[] buffer, int offset, int length) throws IOException {
            this.outputStream.write(buffer, offset, length);
        }

        @Override
        protected Storage toStorage0() throws IOException {
            return new GcsStorage(gcsService, gcsFilename);
        }

        @Override
        public void close() throws IOException {
            super.close();
            this.outputStream.close();
        }
    }
}

Answer 3

I just ran into this problem and looked in the source code for mime4j's Message constructor. It gets the TempFileStorageProvider by calling DefaultStorageProvider.getInstance(). You can change the default to one that doesn't write to the filesystem by calling:

DefaultStorageProvider.setInstance(new MemoryStorageProvider());

That's org.apache.james.mime4j.storage.DefaultStorageProvider.

Thanks for the concise example of using @MultipartForm!

Answer 4

Well, I've found a walk-around for it - I'm using apache commons-upload with RESTEasy, by injecting the HttpServletRequest into a RESTEasy method (and transforming the streams into byte array/string using commons-IO). All packages are app engine supported.

@Path("")
public class Rest {
    @POST
    @Path("/rest/upload")
    public String postContent(@Context HttpServletRequest request) {
        ServletFileUpload upload = new ServletFileUpload();
        FileItemIterator fileIterator = upload.getItemIterator(request);
        while (fileIterator.hasNext()) {
            FileItemStream item = fileIterator.next();
            if ("file".equals(item.getFieldName())){
                byte[] content = IOUtils.toByteArray(item.openStream())
                // Save content into datastore
                // ... 
            } else if ("name".equals(item.getFieldName())){
                String name=IOUtils.toString(item.openStream());
                // Do something with the name string
                // ...
            }
        }
        return "Done";
    } 
}

I would still rather find a RESTEasy solution, to avoid the broil-up code around the fileIterator.

Answer 5

To use the MemoryStorageProvider with RESTEasy you can set the following system property:

-Dorg.apache.james.mime4j.defaultStorageProvider=org.apache.james.mime4j.storage.MemoryStorageProvider

I tried it with RESTEasy 2.3.1.GA and jboss-as-7.1.0.Final.

There also was a bug in prior RESTEasy versions where temporary files where not deleted (https://issues.jboss.org/browse/RESTEASY-681). Using the MemoryStorageProvider is a workaround for that.

Answer 6

I just upgraded resteasy-multipart-provider jar from 2.2.0.GA to 3.1.4.Final . We have to call close method explicitly . It will take care of deleting m4jxxxx.tmp files.

see @docs http://docs.jboss.org/resteasy/docs/3.1.4.Final/userguide/html_single/index.html

package org.jboss.resteasy.plugins.providers.multipart;
public interface MultipartInput {
List<InputPart> getParts();
String getPreamble();
// You must call close to delete any temporary files created
// Otherwise they will be deleted on garbage collection or on JVM exit
void close();
}