SocketTimeoutException while downloading data from

Question

I'm getting a large number of SocketTimeoutException while downloading data from Google Storage. What's the best way to increase the timeout ?

Use case

Download 24 files (10GB total) in parallel from multiple servers using Map/Reduce.

Source Code

  Storage storage = new Storage.Builder(
                new NetHttpTransport(),
                new JacksonFactory(),
                new GoogleCredential().setAccessToken(accessToken))
            .setApplicationName("FooBar")
            .build();

  Storage.Objects.Get getObject = storage.objects().get(bucket, fn);
  getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
  getObject.executeMediaAndDownloadTo(outputStream);

Stack Trace

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.MeteredStream.read(MeteredStream.java:134)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:261)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:209)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
    at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4494)

Edit

This only happens when I download the data from Japan. Everything is fine from US-East and US-West.

Answer 1

Try the following ...

1. Set the proxy in the linux box.

2. While creating storage - new NetHttpTransport() provide your own way to create the socket and add the proxy in the transport.

   public HttpClient myHttpClient() throws Exception {
       SchemeRegistry schemeRegistry = new SchemeRegistry();
       //SetRegisrty for both HTTP and HTTPS - Check google for this.
       schemeRegistry.register(new Scheme("http", PlainSocketFactory
               .getSocketFactory(), YOUR_PROXY_PORT));
       schemeRegistry.register(new Scheme("https", SSLSocketFactory
               .getSocketFactory(), 443));
   
       HttpParams params = new BasicHttpParams();
       HttpConnectionParams.setConnectionTimeout(params, 30 * 1000);  // SET the timeout
       HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
       ClientConnectionManager connManager = new ThreadSafeClientConnManager(
               params, schemeRegistry);
       DefaultHttpClient httpClient = new DefaultHttpClient(connManager,
                   params);
       try {
           int proxyPort = YOUR_PROXY_PORT;
           String proxyHost = "YOUR_PROXT_HOST_NAME";
           if (proxyPort > 0 && proxyHost != null && proxyHost.length() > 0) {
               System.setProperty("https.proxyHost", proxyHost);
               System.setProperty("https.proxyPort", proxyPort + "");
               System.setProperty("http.proxyHost", proxyHost);
               System.setProperty("http.proxyPort", proxyPort + "");
               HttpHost proxy = new HttpHost(proxyHost, proxyPort);
               httpClient.getParams().setParameter(
                       ConnRoutePNames.DEFAULT_PROXY, proxy);
           }
       } catch (NullPointerException e) {
           System.out.println("Proxy error here");
       }
       return httpClient;
   }
   
   public static HttpTransport myNetHttpTransport()
           throws Exception {
       return new ApacheHttpTransport(myHttpClient());
   }
   

Use .setTransport(myNetHttpTransport()) instead of NetHttpTransport().

We spent long time on this. but as of now this seems to be working. Please let me know if any help in this...

Answer 2

Look for a variable called "CommandTimeout", or something of that nature. Usually that determines how long a connection can last before it flags an error.

"ConnectionTimeout" usually means how long the program can try to connect to your source, which isn't really what you need changed, as the connection happens fast, it's the download that is slow.