Puzzled by CookieHandler

Question

I need to access some web pages and pass cookies around the way browsers do. This is easily done using

CookieHandler.setDefault(new MyCookieManager());

but this introduces global state which I need to avoid (Imagine accessing two accounts on the same server concurrently). So what I'd like to do is something like

String doGetWithCookies(URL url, MyCookies myCookies) {
    HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
    myCookies.addToRequest(...);
    myCookies.updateFromResponse(...);
    return getHttpBody(conn);
}

but I can't see how to do it. The methods CookieManager.get and put do accept an URL, but I want to use

• the same cookies with different URLs
• different cookies for the same URL for different accounts

What I've tried: Nothing as there are just four methods available and a single subclass and nothing fits. Parsing the headers manually is surely doable but IMHO no option in 2014. I know about the Apache Http Client, but 1. I'd hope something that trivial needs no half a megabyte library, 2. at the first glance I can't see a solution there either.

Clarification:

Imagine you want to lock to SO as two different users. You can do it by using two computers or two different browsers (Chrome and Firefox) on a single computer. You can't do it in two tabs of a single browser.

What I want is equivalent to the possibility of simulating two browser. In the meantime I've found a related question and posted hacky solution to it.

Still I'm looking for an explanation behind the CookieHandler design.

Answer 1

How are you sending your requests to the server?

If you're using HttpUrlConnection, then you could implement your own cookies. Every time you make a request, add your Cookie:foo=1;bar=2 header to the request. Every time you read a response, check for a Cookie header and store it for subsequent requests.

One tricky part will be knowing when to send a cookie, based on the URL being requested.

There's nothing built-in to the JDK to do this besides CookieHandler, which is lacking.

HttpClient might be a good option, as it has support for granular cookies.

Answer 2

The fundamental problem here, as you mentioned is that

CookieHandler.setDefault(new MyCookieManager());

introduces a global state. Why Oracle failed to provide a simple means of managing cookies on a per session basis is beyond me.

Here's how you can do it, though.

1. Include a URLConnectionCookieManager class with two methods: setCookiesFromCookieJar(urlConnection) and putCookiesInCookieJar(urlConnection).
2. For each separate session, create an instance of the URLConnectionCookieManager class. (Note that its methods are NOT static; they are instance methods.)
3. Before sending a request, call urlConnectionCookieManager.setCookiesFromCookieJar(urlConnection). This method retrieves cookies from the "cookie jar" and adds them to the connection's request headers.
4. After sending a request, call urlConnectionCookieManager.putCookiesInCookieJar(urlConnection). This method retrieves cookies from the connection and puts them in the "cookie jar".

Make sure you use the instance of URLConnectionCookieManager that corresponds to the "session".

The URLConnectionCookieManager class looks like this:

package http;

import java.net.URLConnection;
import java.net.CookieManager;
import java.net.CookieHandler;
import java.net.CookieStore;
import java.net.CookiePolicy;
import java.net.URL;
import java.net.URI;
import java.util.List;
import java.util.Iterator;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.io.IOException;
import java.io.Serializable;
import java.net.URISyntaxException;


public class URLConnectionCookieManager implements Serializable
  {

  public URLConnectionCookieManager()
    {
    this(null, null);
    }

  public URLConnectionCookieManager(
    CookieHandler cookieHandler)
    {
    setCookieHandler(cookieHandler);
    }

  public URLConnectionCookieManager(
    CookieStore cookieStore,
    CookiePolicy cookiePolicy)
    {
    CookieHandler cookieHandler = createCookieHandler(cookieStore, cookiePolicy);
    setCookieHandler(cookieHandler);
    }

  public void putCookiesInCookieJar(
    URLConnection urlConnection) throws IOException
    {
    Map<String, List<String>> headers = urlConnection.getHeaderFields();
    URL url = urlConnection.getURL();

    URI uri = null;
    try
      {
      uri = url.toURI();
      }
    catch (URISyntaxException urise)
      {
      System.out.println("Unable to convert URL to URI while putting cookies in cookie jar.");
      throw new IOException(urise);
      }

    CookieHandler cookieHandler = getCookieHandler();
    cookieHandler.put(uri, headers);
    }

  public void setCookiesFromCookieJar(
    URLConnection urlConnection) throws IOException
    {
    Map<String, List<String>> headerMap = new HashMap<String, List<String>>();

    URL url = urlConnection.getURL();

    URI uri = null;
    try
      {
      uri = url.toURI();
      }
    catch (URISyntaxException urise)
      {
      System.out.println("Unable to convert URL to URI while setting cookies from cookie jar.");
      throw new IOException(urise);
      }

    CookieHandler cookieHandler = getCookieHandler();
    headerMap = cookieHandler.get(uri, headerMap);

    Set<Map.Entry<String, List<String>>> headerSet = headerMap.entrySet();
    Iterator<Map.Entry<String, List<String>>> headerIterator = headerSet.iterator();
    boolean hasNextPair = headerIterator.hasNext();
    while (hasNextPair)
      {
      Map.Entry<String, List<String>> pair = headerIterator.next();
      String key = pair.getKey();
      List<String> cookieList = pair.getValue();

      Iterator<String> cookieIterator = cookieList.iterator();
      boolean hasNextCookie = cookieIterator.hasNext();
      while (hasNextCookie)
        {
        String cookie = cookieIterator.next();
        urlConnection.addRequestProperty(key, cookie);
        hasNextCookie = cookieIterator.hasNext();
        }

      hasNextPair = headerIterator.hasNext();
      }
    }

  public CookieHandler getCookieHandler()
    {
    return this.cookieHandler_;
    }

  protected CookieHandler createCookieHandler(
    CookieStore cookieStore,
    CookiePolicy cookiePolicy)
    {
    CookieHandler cookieHandler = new CookieManager(cookieStore, cookiePolicy);

    return cookieHandler;
    }

  protected void setCookieHandler(
    CookieHandler cookieHandler)
    {
    this.cookieHandler_ = cookieHandler;
    }

  private CookieHandler cookieHandler_;

  }

Answer 3

I actually like your "hacky" solution if you are working with multiple threads and you can guarantee that the same thread is responsible for accessing resources on behalf of the same user. If that is not the case, i.e. two or more threads need to share a user account and the corresponding cookies or a single thread needs to access resources for multiple users, this solution will probably not work...

My first attempt would have been to set an individual CookieManager for every user account:

String doGetForUser(URL url, String username) {
    synchronized (...) {
        CookieManager.setDefault(getCookieManagerForUser(username));
        HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
        // Calling getInputStream on the connection automatically retrieves
        // cookies from the CookieManager and stores new cookies that have been
        // sent from the server.
        return getHttpBody(conn);
    }
}

This is pretty similar to your solution except that it does not rely on the one-user-per-thread assumption. However, locking the CookieManager the entire time isn't probably what you want. But as it turns out, the HttpURLConnection saves a reference to the default CookieManager in its constructor. Exploiting this, we get what I would really call a hacky solution:

String doGetForUser(URL url, String username) {
    HttpsUrlConnection conn;
    synchronized (...) {
        CookieManager.setDefault(getCookieManagerForUser(username));
        // this saves the current CookieManager
        conn = (HttpsURLConnection) url.openConnection();
        CookieManager.setDefault( ... ); // restore original one?
    }
    return getHttpBody(conn);
}

Now we "only" lock the CookieManager for the connection setup which will probably increase concurrency. But it's still pretty ugly and you would now have to make sure that you lock the CookieManager if you make requests from somewhere else that should not use user-specific cookies...

I skimmed the source code of HttpURLConnection and HttpClient to see when it actually retrieves and stores cookies. Apparently, the only place where the CookieManager is queried for the cookies that should be sent is in the private setCookieHeader method, which is called by both getOutputStream and getInputStream before the request is sent. setCookieHeader passes the current request headers to the installed CookieManager. Maybe we could use these headers instead?

String doGetForUser(URL url, String username) {
    HttpURLConnection conn = (HttpsURLConnection) url.openConnection();
    conn.setRequestProperty("X-Username", username);
    return getHttpBody(conn);
}

class UsernameCookieHandler extends CookieHandler {
    @Override
    public Map<String, List<String>>get(URI uri, Map<String, List<String>> requestHeaders) throws IOException {
        if (requestHeaders.containsKey("X-Username"))
            return getUserSpecificCookies(uri, requestHeaders);
        else
            return getCommonCookies(uri, requestHeaders);
    }

    private Map<String, List<String>> getUserSpecificCookies(URI uri, Map<String, List<String>> requestHeaders) {
        // evaluate X-Username and get cookies from a special CookieStore or so...
    }

    private Map<String, List<String>> getCommonCookies(URI uri, Map<String, List<String>> requestHeaders) {
        // get cookies from a common CookieStore...
    }

    @Override
    public void put(URI uri, Map<String, List<String>> responseHeaders) throws IOException {
        // ???
    }
}

This one retrieves user-specific cookies if there is an application-defined request header X-Username and gets a common set of cookies if no such header exists. However, updating the user-specific cookies won't be this easy because the server will most certainly not send the X-Username header back to us. The idea would of course be to somehow determine the username from the responseHeaders. But I currently don't see a way how to inject the correct header field in the server`s response without setting up a proxy server.... Sorry :(

---

FYI: The only place I could find that calls CookieManager.put is the private parseHTTPHeader method in HttpClient. This method is of course invoked from getInputStream before you can read the response body.

Answer 4

In that case you need to hack the request reader so it won't use the standard "sessionId" parameter but your own "tabSessionId". This should be done by overriding the header parsing method. Try overriding java.net.URLConnection.getHeaderField(int) to use your own cookie when "sessionId" is requested