a problem about urlfetch over 1M in app engine

Question

to fetch() over 1M in app engine,i use the range header and then combine those pieces.and my codes:

int startpos=0;
int endpos;
int seg=1;
int len=1;
while(len>0){
endpos=startpos+seg;
httpConn = (HttpURLConnection) u.openConnection();
httpConn.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14");

con.setRequestProperty("Range", "bytes=" + startpos + "-" + endpos);  
con.connect();
InputStream in=con.getInputStream();

len=con.getContentLength();
byte[] b=new byte[len];
in.read(b, 0, len);

startpos+=len; 

} but when it goes to the "InputStream in=con.getInputStream();",its debug is " URL Fetch Response too large problems" so i don't know what the wrong with these codes. and there are other ways to fetch() over 1M?

Answer 1

I had the same problem and hacked up a little class to simulate an input stream on Appengine using the HTTP range parameter. It allows you to read files bigger then the limit in a line-oriented fashion. I am attaching it below, although you may need to adapt it for your purposes:

package com.theodorebook.AEStreamer;

import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Arrays;
import java.util.logging.Logger;

/**
 * A class to simulate a stream in appengine, which insists on downloading
 * an entire URL before letting you do anything with it.  This enables one
 * to read files larger than the size limits.
 * 
 * @author Theodore Book (theodorebook at gmail dot com)
 *
 */
public class AEStreamer {
    private static final int BITE_SIZE = 0x10000;   //How big a chunk to grab at a time
    private static final byte TERMINATOR = '\n';    //String terminator

    private int mCurrentPosition = 0;   //The current position in the file
    private int mOffset = -1;   //The offset of the current block
    private long mValidBytes = 0;   //The number of valid bytes in the chunk
    private byte[] mChunk = new byte[BITE_SIZE];
    private boolean mComplete = false;
    private String mURL;

    private static final Logger log = Logger.getLogger(AEStreamer.class.getName());

    public AEStreamer(String url) {
        mURL = url;
    }

    /**
     * Returns the next line from the source, or null on empty
     * @return
     */
    public String readLine() {
        String line = "";

        //See if we have something to read
        if (mCurrentPosition >= mOffset + mValidBytes) {
            if (mComplete)
                return null;
            readChunk();
        }
        if (mValidBytes == 0)
            return null;

        //Read until we reach a terminator
        int endPtr = mCurrentPosition - mOffset;
        while (mChunk[endPtr] != TERMINATOR) {
            endPtr++;

            //If we reach the end of the block
            if (endPtr == mValidBytes) {
                line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
                mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
                if (mComplete) {
                    return line;
                } else {
                    readChunk();
                    endPtr = mCurrentPosition - mOffset;
                }
            }
        }
        line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
        mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
        mCurrentPosition++;
        return line;
    }

    /**
     * Reads the next chunk from the server
     */
    private void readChunk() {
        if (mOffset < 0)
            mOffset = 0;
        else
            mOffset += BITE_SIZE;

        try {
            URL url = new URL(mURL);
            URLConnection request = url.openConnection();
            request.setRequestProperty("Range", "bytes=" + (mOffset + 1) + "-" + (mOffset + BITE_SIZE)); 
            InputStream inStream = request.getInputStream();
            mValidBytes = inStream.read(mChunk);
            inStream.close();
        } catch (Exception e) {
            log.severe("Unable to read " + mURL + ": " + e.getLocalizedMessage());
            mComplete = true;
            mValidBytes = 0;
            return;
        }

        if (mValidBytes < BITE_SIZE)
            mComplete = true;

        //log.info("Read " + mValidBytes + " bytes");
    }
}

Answer 2

Not all HTTP servers support range requests, especially when it comes to frameworks serving dynamic content - they'll simply ignore the Range header and send you the whole response.

The recent release of 1.4.0 increased the URLFetch response limit to 32MB, though, so you no longer need to do this.