How to login in web site using Java

Question

I want to access some pages of web site https://myoffice.bt.com which requires user authentication using java. We have to sign in first to access pages. I have wriiten following code.

package root;

import java.io.IOException;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;


public class Url
{
 public static void main(String[] args) throws IOException
 {
  HttpClient client = new HttpClient();

  client.getParams().setParameter(
      HttpMethodParams.USER_AGENT,
      "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"
  );

  client.getState().setCredentials(
     new AuthScope("https://myoffice.bt.com", 443,  AuthScope.ANY_REALM),
     new UsernamePasswordCredentials("username", "password")  ); 

  PostMethod get = new PostMethod("https://myoffice.bt.com/youraccount/default.aspx");
  get.setDoAuthentication( true );
  System.out.println(get.getFollowRedirects());
  //get.setFollowRedirects(true);


  try {
    // execute the GET
     int status = client.executeMethod( get );

     // print the status and response
     System.out.println(status + "\n" + get.getResponseBodyAsString());

     } finally {
     // release any connection resources used by the method
      get.releaseConnection();
     } 


 }

}

But it gives following errors.

> Jun 22, 2010 12:14:40 PM org.apache.commons.httpclient.HttpMethodDirector isRedirectNeeded
INFO: Redirect requested but followRedirects is disabled
302

If I uncomment get.setFollowingRedirects line, It gives another error.

Exception in thread "main" java.lang.IllegalArgumentException: Entity enclosing requests cannot be redirected without user intervention
 at org.apache.commons.httpclient.methods.EntityEnclosingMethod.setFollowRedirects(Unknown Source)
 at root.Url.main(Url.java:30)

Can any one help me here? Can we do form based authentication using HttpClient?

Thanks.

Answer 1

Java Version : Works well with siteminder protected resources, tested with commons httpClient 4.3.3

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.List;

import javax.net.ssl.SSLContext;

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContexts;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.client.LaxRedirectStrategy;
import org.apache.http.util.EntityUtils;

public class AccessSiteminderProtectedResource {

    private static final String PASSWORD = "pwd";
    private static final String USER_NAME = "userId";
    private static final String SITEMINDER_PROTECTED_RESOURCE = "protectedResource";
    private static final String SITEMINDER_LOGIN_URL = "siteMinderLoginUrl?TARGET=-SM-" + SITEMINDER_PROTECTED_RESOURCE;

    public static void main(String[] args) throws Exception {

        BasicCookieStore cookieStore = new BasicCookieStore();

        SSLContext sslcontext = buildSSLContext();

        SSLConnectionSocketFactory sslsf = buildSSLConnectionSocketFactory(sslcontext);

        CloseableHttpClient httpclient = buildHttpClient(cookieStore, sslsf);

        try {

            String nextLocation = executeLogin(cookieStore, httpclient);

            accessApp(httpclient, nextLocation);

        } finally {
            httpclient.close();
        }
    }

    private static SSLContext buildSSLContext()
            throws NoSuchAlgorithmException, KeyManagementException,
            KeyStoreException {
        SSLContext sslcontext = SSLContexts.custom()
                .setSecureRandom(new SecureRandom())
                .loadTrustMaterial(null, new TrustSelfSignedStrategy()).build();
        return sslcontext;
    }

    private static SSLConnectionSocketFactory buildSSLConnectionSocketFactory(
            SSLContext sslcontext) {
        SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
                sslcontext,
                SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
        return sslsf;
    }

    private static CloseableHttpClient buildHttpClient(
            BasicCookieStore cookieStore, SSLConnectionSocketFactory sslsf) {
        CloseableHttpClient httpclient = HttpClients.custom()
                .setSSLSocketFactory(sslsf).setDefaultCookieStore(cookieStore)
                .setRedirectStrategy(new LaxRedirectStrategy())
                .build();
        return httpclient;
    }

    private static String executeLogin(BasicCookieStore cookieStore,
            CloseableHttpClient httpclient) throws URISyntaxException,
            IOException, ClientProtocolException {

        HttpUriRequest loginPost = RequestBuilder
                .post()
                .setUri(new URI(SITEMINDER_LOGIN_URL))
                .addParameter("USER", USER_NAME)
                .addParameter("PASSWORD", PASSWORD).build();

        System.out.println("executing request" + loginPost.getRequestLine() + "\n");

        CloseableHttpResponse loginResponse = httpclient.execute(loginPost);
        String nexLocation;
        try {
            HttpEntity loginResponseEntity = loginResponse.getEntity();

            System.out.println("Login form post Status: " + loginResponse.getStatusLine());
            EntityUtils.consume(loginResponseEntity);
            System.out.println();

            System.out.println("Post logon cookies:");
            System.out.println();
            displayCookies(cookieStore);
            System.out.println();
            System.out.println();

            System.out.println("Login Post Headers----------------------------------------");
            displayHeaders(loginResponse);

            System.out.println();
            System.out.println();

            nexLocation = SITEMINDER_PROTECTED_RESOURCE;
        } finally {
            loginResponse.close();
        }

        return nexLocation;
    }

    private static void accessApp(CloseableHttpClient httpclient, String nextLocation) throws IOException, ClientProtocolException {
        HttpGet appGet = new HttpGet(nextLocation);

        System.out.println("executing request" + appGet.getRequestLine());

        CloseableHttpResponse response = httpclient.execute(appGet);
        try {
            HttpEntity entity = response.getEntity();

            System.out.println("\n\n\n\n---------------------------------------- \n");

            System.out.println("App Get Status: " + response.getStatusLine());
            System.out.println(EntityUtils.toString(entity));
            EntityUtils.consume(entity);

        } finally {
            response.close();
        }
    }

    private static void displayHeaders(CloseableHttpResponse loginResponse) {
        for (Header header : loginResponse.getAllHeaders()) {
            System.out.println(header);
        }
    }

    private static void displayCookies(BasicCookieStore cookieStore) {
        List<Cookie> cookies = cookieStore.getCookies();
        if (cookies.isEmpty()) {
            System.out.println("None");
        } else {
            for (int i = 0; i < cookies.size(); i++) {
                System.out.println("- " + cookies.get(i).toString());
            }
        }
    }

}

Answer 2

First - please don't name your PostMethod variable get.

Second, try this:

PostMethod post = new PostMethod("yourUrl")
{
    @Override
    public boolean getFollowRedirects()
    {
        return true;
    }
};

If you ever happen to be on the "other side" and want to prevent your users from suffering, use the response code 303 (See Other) when redirecting a POST request to a GET, instead of the common 302 and 301 (per RFC). Regular browsers tend to be nice, break the rules and NOT ask us to confirm these redirects, but a lot of mobile browsers still do.

Regarding your question about form based authentication - you just need to figure out the parameter names to use (by looking at the source of the website where you "normally" log in, for example), and then populate them with the appropriate values:

post.addParameter("username", username);
post.addParameter("password", password);

---

I played around with the login form at myoffice.bt.com, there's a few things going on in JavaScript.

The form is submitted to https://myoffice.bt.com/siteminderagent/forms/login.fcc

The form elements that are submitted were as follows (name=value, some values were empty):

Segment=btb.hub
SubSegment=
searchType=0
searchPlatform=BEA
lob=btb.hub
queryText=
searchText=
ctl00$masterWebpartManager$gwpCustomLogin1$CustomLogin1$UserName=[e-mail]
ctl00$masterWebpartManager$gwpCustomLogin1$CustomLogin1$PWD=yourpwd
ctl00$masterWebpartManager$gwpCustomLogin1$CustomLogin1$RememberMe=on
USER=[e-mail]
PASSWORD=yourpwd
SMENC=ISO-8859-1
SMLOCALE=US-EN
userFirstLoginUrl=https://myoffice.bt.com/ManageBusinessApplications/SecretQA.aspx
PrivateLoginSuccessUrl=https://myoffice.bt.com/sm/privatecreatesession.aspx?siteArea=btb.mya
PublicLoginSuccessUrl=https://myoffice.bt.com/sm/createsession.aspx?siteArea=btb.mya
target=https://myoffice.bt.com/sm/privatecreatesession.aspx?siteArea=btb.mya&TARGET=https%3a%2f%2fmyoffice.bt.com%2fdefault.aspx (hidden)
submitStatus=
smauthreason=
smagentname=
postpreservationdata=
AnonUserName=[e-mail]
authMode=SITEMINDER
smUrl=https://myoffice.bt.com/siteminderagent/forms/login.fcc
notSMUrl=https://myoffice.bt.com/default.aspx
smIdentifier=1

Try adding some or all of these (at least USER and PASSWORD) to your PostMethod, and make sure you are submitting to the correct URL.

Answer 3

If that website uses Siteminder authentication you will not be able to log in just like that. Siteminder uses cookies to identify authenticated sessions. These cookies are valid only as long as your session is alive. If you are not logged in, then the server redirects you to the Siteminder login page (hence the redirect). So what you'll need to do is follow the redirect, send your credentials (username/password), then follow the redirect again, sending the received cookies.

I've recorded sessions for regression tests using The Grinder (http://grinder.sourceforge.net/), and it was able to log in to the Siteminder protected website automatically! So it is definitely possible, but you'll have to do a bit more than simply send an HTTP request...

The best solution would be some kind of other authentication, like certificate-based authentication (but of course this must be configured on the server side as well, so this may not be an option in this case). Why not ask BT whether they provide other authentication methods?

EDIT: I've just found this: http://www.codeproject.com/KB/IP/SiteminderHttpWebRequest.aspx The source code is in VB, but the article is excellent, and it should be no problem to translate VB code to Java... ;-)