Trying to access hosted SharePoint 2013 from Java

Question

I'm trying to access our company SharePoint 2013 instance, which is hosted at ourinstance.sharepoint.com, with small Java web application.

I registered application through _layouts/15/AppRegNew.aspx this way:

1. I let SP generate Client ID, Client Secret,
2. to App Domain I set: ourdomain:8443
3. to Redirect URL I set: https://ourdomain:8443/our-redirect-processing

I edited application permissions with _layouts/15/appinv.aspx, where I looked it up by client ID and edited its Permission Request XML to contain:

<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://insideidc.sharepoint.com/oursite/web" Right="FullControl"/>
</AppPermissionRequests>

Handler behind https://ourdomain:8443/our-redirect-processing is doing this:

JsonWebSignature jws = JsonWebSignature.parse(new JacksonFactory(), request.getParameter("SPAppToken"));
JsonParser jsonParser = new JacksonFactory().createJsonParser(jws.getPayload().get("appctx").toString());
AppCtx appCtx = jsonParser.parse(AppCtx.class, new CustomizeJsonParser());
String appctxsender=jws.getPayload().get("appctxsender").toString();
String[] splitApptxSender = appctxsender.split("@");
String sharepointServerHostName = new URL(request.getParameter("SPSiteUrl")).getHost();
String resource = splitApptxSender[0] + "/" + sharepointServerHostName + "@" + splitApptxSender[1];
AuthorizationCodeTokenRequest tokenRequest = new AuthorizationCodeTokenRequest(new NetHttpTransport(), new JacksonFactory(),
    new GenericUrl(appCtx.getSecurityTokenServiceUri()), jws.getPayload().get("refreshtoken").toString());
tokenRequest.setRedirectUri(request.getRequestURL().toString());
String aud = (String) jws.getPayload().getAudience();
tokenRequest.setClientAuthentication(new ClientParametersAuthentication(aud, secrets.get(aud)));
tokenRequest.setGrantType("refresh_token");
tokenRequest.set("resource", resource);
tokenRequest.set("refresh_token", jws.getPayload().get("refreshtoken").toString());
TokenResponse response = tokenRequest.execute();
token = response.getAccessToken();

It uses com.google.api.client. auth, http and json classes.

The token I get I use in REST call to this URL:

https://ourinstance.sharepoint.com/oursite/_api/web/getFolderByServerRelativeUrl('/thefolderIwant')/Files

with these Headers:

Accept: application/json;odata=verbose
Authorization: Bearer theToken

The response asks me for logging in, while response header has WWW-Authenticate: NTLM set.

First question: should such complex process of obtaining OAuth token end up with another credentials request?

Second and main question: how can I construct domain\username for NTLM Authenticator, which I can build, when SharePoint is hosted for us?

Answer 1

I had a similar experience when trying to access Microsoft's Project Online. I found some helpful information from AllThatJS which pointed me in the right direction. He suggested sniffing the packets using Fiddler. Once I did that, I saw what was actually going on. Here is some Java code I used to solve this problem, using Apache's HttpClient, Apache's common-io, and log4j to solve this :

/**

    This is for using Java to connect with Microsoft's Project Online

    If you go into your Project Online, go to 'Server Settings' -> 'Manage Users', and look under the column 'User Logon Account'

    If it looks like :
        i:0#.w|domain\\username
    then you can just scroll down to where I call 

                httpclient.getCredentialsProvider().setCredentials(
                        new AuthScope(url.getHost(), url.getPort(), AuthScope.ANY_REALM, AuthScope.ANY_SCHEME),
                        new NTCredentials(
                                USERNAME,
                                PASSWORD,
                                url.getHost(), 
                                DOMAIN));


    However, if it looks more like :
        i:0#.f|membership|[e-mail]
    then you'll need to use OAuth, which is what this file demonstrates.

*/

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.Calendar;
import java.util.UUID;


import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;

import org.apache.commons.io.IOUtils;// from commons-io-2.4.jar
import org.apache.http.Header;// from httpcore-4.2.4.jar
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.NTCredentials;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.log4j.Logger;// from log4j.jar
import org.w3c.dom.Document;
import org.xml.sax.SAXException;

public class ProjectOnline {

    private static final Logger logger         = Logger.getLogger(ProjectOnline.class);

    private static final String USERNAME       = "your user's email address";
    private static final String PASSWORD       = "password";
    private static final String DOMAIN         = "YOURDOMAIN";
    private static final String PROJECT_SERVER = "https://your.project.online.sharepoint.com";
    private static final String EMAIL_DOMAIN   = "@your.company.com";


    public static void main(String[] args) {

        ProjectOnline dao = new ProjectOnline();
        System.out.println(dao.getOAuthCookie());

    }

    private boolean needToUseOAuth(String username) {
        if (username == null) return false;
        return username.toLowerCase().endsWith(EMAIL_DOMAIN.toLowerCase());
    }

    public String getOAuthCookie() {

        if (needToUseOAuth(USERNAME)) {

            String samlSecurityToken = postLoginCredentialsToOurOAuthService();

            if (samlSecurityToken != null && samlSecurityToken.isEmpty() == false) {

                String binarySecurityToken = postSamlSecurityTokenToSecondSite(samlSecurityToken);

                if (binarySecurityToken != null && binarySecurityToken.isEmpty() == false) {

                    String spoidcrlCookie = getSpoidcrlCookie(binarySecurityToken);
                    return spoidcrlCookie;
                } else {
                    //System.out.println("getXMLDocument - OAuth authentication / authorization failed : Binary Security Token was not found");
                    logger.error("getOAuthCookie - OAuth authentication / authorization failed : Binary Security Token was not found");
                }
            } else {
                //System.out.println("getXMLDocument - OAuth authentication / authorization failed : SAML Security Token was not found");
                logger.error("getOAuthCookie - OAuth authentication / authorization failed : SAML Security Token was not found");
            }
        }

        return "";
    }

    // Step 1 - Find the URL to your company's OAuth site
    private String getOurOAuthServerURL(String emailAddress) {

        DefaultHttpClient httpclient = new DefaultHttpClient();

        // Go to this site, passing in email address. Should tell the URL for your company's OAuth site
        HttpPost httppost = new HttpPost("https://login.microsoftonline.com/GetUserRealm.srf?xml=1&login=" + emailAddress);

        try {

            HttpResponse response = httpclient.execute(httppost);
            HttpEntity entity = response.getEntity();

            String status = response.getStatusLine().toString();
            //System.out.println("getOurOAuthServerURL - status = " + status);
            logger.info("getOurOAuthServerURL - status = " + status);

            // If response status doesn't equal 'OK' then it didn't work.
            if (entity != null && "HTTP/1.1 200 OK".equals(status)) {

                StringWriter writer = new StringWriter();
                IOUtils.copy(entity.getContent(), writer, "utf-8");
                String xml = writer.toString();

                //System.out.println(xml);
                //logger.debug(xml);

                String ourAuthURL = getSTSAuthURL(xml);
                //System.out.println("ourAuthURL = " + ourAuthURL);
                return ourAuthURL;

            }
        } catch (UnsupportedEncodingException e) {
            logger.error("getOurOAuthServerURL ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (ClientProtocolException e) {
            logger.error("getOurOAuthServerURL ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IllegalStateException e) {
            logger.error("getOurOAuthServerURL ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IOException e) {
            logger.error("getOurOAuthServerURL ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } finally {
            // When HttpClient instance is no longer needed,
            // shut down the connection manager to ensure
            // immediate deallocation of all system resources
            httpclient.getConnectionManager().shutdown();
        }
        return "";
    }

    private String getBinarySecurityToken(String xml) {
        return getValuesBetweenTags(xml, "<wsse:BinarySecurityToken Id=\"Compact0\">", "</wsse:BinarySecurityToken>");
    }

    private String getSTSAuthURL(String xml) {
        return getValuesBetweenTags(xml, "<STSAuthURL>", "</STSAuthURL>");
    }

    private String getSamlSecurityToken(String xml) {
        return getValuesBetweenTags(xml, "<t:RequestedSecurityToken>", "</t:RequestedSecurityToken>");
    }

    private String getValuesBetweenTags(String xml, String tagToLeft, String tagToRight) {
        if (xml == null || xml.isEmpty()) return "";

        int startToken = xml.indexOf(tagToLeft);
        if (startToken > -1) {
            return xml.substring(startToken + tagToLeft.length(), xml.indexOf(tagToRight, startToken));
        } else {
            return "";
        }
    }

    private String getTimeString(int minutesInFuture) {
        Calendar rightNow = Calendar.getInstance();
        rightNow.add(Calendar.SECOND, (((rightNow.get(Calendar.ZONE_OFFSET) + (rightNow.get(Calendar.DST_OFFSET)))/-1000)));
        rightNow.add(Calendar.MINUTE, minutesInFuture);
        String timeString = String.format("%d-%02d-%02dT%02d:%02d:%02d.0000000Z", 
                                     rightNow.get(Calendar.YEAR), 
                                    (rightNow.get(Calendar.MONTH) + 1), 
                                     rightNow.get(Calendar.DATE), 
                                     rightNow.get(Calendar.HOUR_OF_DAY),
                                     rightNow.get(Calendar.MINUTE),
                                     rightNow.get(Calendar.SECOND));
        return timeString;
    }

    // Step 2 - POST an XML message, with a few key fields filled in (rest can be left as-is)
    // This should be sent to your company's OAuth site
    private String postLoginCredentialsToOurOAuthService() {

        String ourOAuthService = getOurOAuthServerURL(USERNAME);

        DefaultHttpClient httpclient = new DefaultHttpClient();

        StringBuilder xmlString = new StringBuilder();

        xmlString.append("<s:Envelope xmlns:s=\"http://www.w3.org/2003/05/soap-envelope\" ");
        xmlString.append("xmlns:wsse=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\"  ");
        xmlString.append("xmlns:saml=\"urn:oasis:names:tc:SAML:1.0:assertion\"  ");
        xmlString.append("xmlns:wsp=\"http://schemas.xmlsoap.org/ws/2004/09/policy\"  ");
        xmlString.append("xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\"  ");
        xmlString.append("xmlns:wsa=\"http://www.w3.org/2005/08/addressing\"  ");
        xmlString.append("xmlns:wssc=\"http://schemas.xmlsoap.org/ws/2005/02/sc\"  ");
        xmlString.append("xmlns:wst=\"http://schemas.xmlsoap.org/ws/2005/02/trust\"> ");
        xmlString.append("<s:Header> ");
        xmlString.append("<wsa:Action s:mustUnderstand=\"1\">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</wsa:Action> ");
        xmlString.append("<wsa:To s:mustUnderstand=\"1\">" + ourOAuthService + "</wsa:To> ");
        xmlString.append("<wsa:MessageID>").append(UUID.randomUUID().toString()).append("</wsa:MessageID> ");
        xmlString.append("<ps:AuthInfo xmlns:ps=\"http://schemas.microsoft.com/Passport/SoapServices/PPCRL\" Id=\"PPAuthInfo\"> ");
        xmlString.append("<ps:HostingApp>Managed IDCRL</ps:HostingApp> ");
        xmlString.append("<ps:BinaryVersion>6</ps:BinaryVersion> ");
        xmlString.append("<ps:UIVersion>1</ps:UIVersion> ");
        xmlString.append("<ps:Cookies></ps:Cookies> ");
        xmlString.append("<ps:RequestParams>AQAAAAIAAABsYwQAAAAxMDMz</ps:RequestParams> ");
        xmlString.append("</ps:AuthInfo> ");
        xmlString.append("<wsse:Security> ");
        xmlString.append("<wsse:UsernameToken wsu:Id=\"user\"> ");
        xmlString.append("<wsse:Username>").append(USERNAME).append("</wsse:Username> ");
        xmlString.append("<wsse:Password>").append(PASSWORD).append("</wsse:Password> ");
        xmlString.append("</wsse:UsernameToken> ");
        xmlString.append("<wsu:Timestamp Id=\"Timestamp\"> ");
        xmlString.append("<wsu:Created>" + getTimeString(0) + "</wsu:Created> ");
        xmlString.append("<wsu:Expires>" + getTimeString(10) + "</wsu:Expires> ");
        xmlString.append("</wsu:Timestamp> ");
        xmlString.append("</wsse:Security> ");
        xmlString.append("</s:Header> ");
        xmlString.append("<s:Body> ");
        xmlString.append("<wst:RequestSecurityToken Id=\"RST0\"> ");
        xmlString.append("<wst:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</wst:RequestType> ");
        xmlString.append("<wsp:AppliesTo> ");
        xmlString.append("<wsa:EndpointReference> ");
        xmlString.append("<wsa:Address>urn:federation:MicrosoftOnline</wsa:Address> ");
        xmlString.append("</wsa:EndpointReference> ");
        xmlString.append("</wsp:AppliesTo> ");
        xmlString.append("<wst:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</wst:KeyType> ");
        xmlString.append("</wst:RequestSecurityToken> ");
        xmlString.append("</s:Body> ");
        xmlString.append("</s:Envelope> ");

        HttpPost httppost = new HttpPost(ourOAuthService);

        try {

            httppost.addHeader("Content-Type", "application/soap+xml; charset=utf-8");

            httppost.setEntity(new StringEntity(xmlString.toString()));// Set this in the body

            HttpResponse response = httpclient.execute(httppost);
            HttpEntity entity = response.getEntity();

            String status = response.getStatusLine().toString();
            //System.out.println("postLoginCredentialsToOurOAuthService - status = " + status);
            logger.info("postLoginCredentialsToOurOAuthService - status = " + status);

            // If response status doesn't equal 'OK' then it didn't work.
            if (entity != null && "HTTP/1.1 200 OK".equals(status)) {

                StringWriter writer = new StringWriter();
                IOUtils.copy(entity.getContent(), writer, "utf-8");
                String xml = writer.toString();

                //System.out.println(xml);
                //logger.debug(xml);

                // Now, extract out the SAML Security Token. It is several lines (~49, if you parse it out).
                String samlSecurityToken = getSamlSecurityToken(xml);
                //System.out.println("samlSecurityToken = " + samlSecurityToken);
                return samlSecurityToken;

            }
        } catch (UnsupportedEncodingException e) {
            logger.error("postLoginCredentialsToOurOAuthService ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (ClientProtocolException e) {
            logger.error("postLoginCredentialsToOurOAuthService ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IllegalStateException e) {
            logger.error("postLoginCredentialsToOurOAuthService ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IOException e) {
            logger.error("postLoginCredentialsToOurOAuthService ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } finally {
            // When HttpClient instance is no longer needed,
            // shut down the connection manager to ensure
            // immediate deallocation of all system resources
            httpclient.getConnectionManager().shutdown();
        }
        return "";
    }

    // Step 3 - Now that you have the SAML Security Token, you embed it within some other generic XML, and send back to Microsoft server
    private String postSamlSecurityTokenToSecondSite(String samlSecurityToken) {

        DefaultHttpClient httpclient = new DefaultHttpClient();

        try {

            HttpPost httppost = new HttpPost("https://login.microsoftonline.com/rst2.srf");

            httppost.addHeader("User-Agent", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)");

            StringBuilder xmlString = new StringBuilder();

            xmlString.append("<S:Envelope xmlns:S=\"http://www.w3.org/2003/05/soap-envelope\" ");
            xmlString.append("xmlns:wsse=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\" ");
            xmlString.append("xmlns:wsp=\"http://schemas.xmlsoap.org/ws/2004/09/policy\" ");
            xmlString.append("xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\" ");
            xmlString.append("xmlns:wsa=\"http://www.w3.org/2005/08/addressing\" ");
            xmlString.append("xmlns:wst=\"http://schemas.xmlsoap.org/ws/2005/02/trust\">");
            xmlString.append("<S:Header>");
            xmlString.append("<wsa:Action S:mustUnderstand=\"1\">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</wsa:Action>");
            xmlString.append("<wsa:To S:mustUnderstand=\"1\">https://login.microsoftonline.com/rst2.srf</wsa:To>");
            xmlString.append("<ps:AuthInfo xmlns:ps=\"http://schemas.microsoft.com/LiveID/SoapServices/v1\" Id=\"PPAuthInfo\">");
            xmlString.append("<ps:BinaryVersion>5</ps:BinaryVersion>");
            xmlString.append("<ps:HostingApp>Managed IDCRL</ps:HostingApp>");
            xmlString.append("</ps:AuthInfo>");
            xmlString.append("<wsse:Security>");
            xmlString.append(samlSecurityToken);
            xmlString.append("</wsse:Security>");
            xmlString.append("</S:Header>");
            xmlString.append("<S:Body>");
            xmlString.append("<wst:RequestSecurityToken xmlns:wst=\"http://schemas.xmlsoap.org/ws/2005/02/trust\" Id=\"RST0\">");
            xmlString.append("<wst:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</wst:RequestType>");
            xmlString.append("<wsp:AppliesTo>");
            xmlString.append("<wsa:EndpointReference>");
            xmlString.append("<wsa:Address>sharepoint.com</wsa:Address>");
            xmlString.append("</wsa:EndpointReference>");
            xmlString.append("</wsp:AppliesTo>");
            xmlString.append("<wsp:PolicyReference URI=\"MBI\"></wsp:PolicyReference>");
            xmlString.append("</wst:RequestSecurityToken>");
            xmlString.append("</S:Body>");
            xmlString.append("</S:Envelope>");

            httppost.addHeader("Content-Type", "application/soap+xml; charset=utf-8");

            httppost.setEntity(new StringEntity(xmlString.toString()));

            HttpResponse response = httpclient.execute(httppost);
            HttpEntity entity = response.getEntity();

            String status = response.getStatusLine().toString();
            //System.out.println("postSamlSecurityTokenToSecondSite - status = " + status);
            logger.info("postSamlSecurityTokenToSecondSite - status = " + status);

            // If response status doesn't equal 'OK' then it didn't work.
            if (entity != null && "HTTP/1.1 200 OK".equals(status)) {

                StringWriter writer = new StringWriter();
                IOUtils.copy(entity.getContent(), writer, "utf-8");
                String xml = writer.toString();

                //System.out.println(xml);
                //logger.debug(xml);

                // Extract out the value from just one, single line of this returned XML file
                String binarySecurityToken = getBinarySecurityToken(xml);
                //System.out.println("binarySecurityToken = " + binarySecurityToken);
                return binarySecurityToken;

            }
        } catch (UnsupportedEncodingException e) {
            logger.error("postSamlSecurityTokenToSecondSite ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (ClientProtocolException e) {
            logger.error("postSamlSecurityTokenToSecondSite ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IllegalStateException e) {
            logger.error("postSamlSecurityTokenToSecondSite ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IOException e) {
            logger.error("postSamlSecurityTokenToSecondSite ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } finally {
            // When HttpClient instance is no longer needed,
            // shut down the connection manager to ensure
            // immediate deallocation of all system resources
            httpclient.getConnectionManager().shutdown();
        }

        return "";
    }

    // Step 4 - Using the extracted value, the Binary Security Token, build a header, and add it to the next request
    // This will go to your own Project Online server, which should return back a reply message, containing a 'Set-Cookie' cookie with 'SPOIDCRL' in the value
    private String getSpoidcrlCookie(String binarySecurityToken) {
        try {

            HttpGet httpget = new HttpGet(PROJECT_SERVER + "/_vti_bin/idcrl.svc/");

            httpget.addHeader("Authorization", "BPOSIDCRL " + binarySecurityToken);

            DefaultHttpClient httpclient = new DefaultHttpClient();
            HttpResponse response = httpclient.execute(httpget);

            String status = response.getStatusLine().toString();
            //System.out.println("getSpoidcrlCookie - status = " + status);
            logger.info("getSpoidcrlCookie - status = " + status);

            // If response status doesn't equal 'OK' then it didn't work.
            if ("HTTP/1.1 200 OK".equals(status)) {

                Header[] headers = response.getHeaders("Set-Cookie");
                for (Header header : headers) {
                    if (header.getValue().contains("SPOIDCRL")) {
                        String spoidcrlCookie = header.getValue();
                        //System.out.println("Found SPOIDCRL cookie : " + spoidcrlCookie);
                        return spoidcrlCookie;
                    }
                }
            }
        } catch (ClientProtocolException e) {
            logger.error("getSpoidcrlCookie ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IOException e) {
            logger.error("getSpoidcrlCookie ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (Exception e) {
            logger.error("getSpoidcrlCookie ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        }
        return null;
    }

    public Document getXMLDocument(String strURL) {

        String spoidcrlCookie = getOAuthCookie();

        DefaultHttpClient httpclient = new DefaultHttpClient();

        HttpGet httpget = new HttpGet(strURL);

        try {

            URL url = new URL(strURL);

            if (needToUseOAuth(USERNAME)) {

                httpget.addHeader("Cookie", spoidcrlCookie);

            } else {

                // Otherwise, can just use this simple way of logging in, using the Domain
                httpclient.getCredentialsProvider().setCredentials(
                        new AuthScope(url.getHost(), url.getPort(), AuthScope.ANY_REALM, AuthScope.ANY_SCHEME),
                        new NTCredentials(
                                USERNAME,
                                PASSWORD,
                                url.getHost(), 
                                DOMAIN));

            }

            //System.out.println("getXMLDocument - strURL " + strURL);
            //logger.info("getXMLDocument - strURL " + strURL);
            //logger.info("getXMLDocument - executing request " + httpget.getRequestLine());
            HttpResponse response = httpclient.execute(httpget);//httppost
            HttpEntity entity = response.getEntity();

            //logger.info("getXMLDocument - ----------------------------------------");
            //for (Header header : response.getAllHeaders()) {
            //  System.out.println("getXMLDocument - header = " + header.toString());
            //}
            String status = response.getStatusLine().toString();
            //System.out.println("getXMLDocument - status = " + status);
            //logger.info("getXMLDocument - status = " + status);

            // If response status doesn't equal 'OK' then it didn't work.
            if (entity != null && "HTTP/1.1 200 OK".equals(status)) {

                DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
                DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();

                StringWriter writer = new StringWriter();
                IOUtils.copy(entity.getContent(), writer, "utf-8");
                String xml = writer.toString();

                //System.out.println(xml);
                //logger.debug(xml);

                if (xml.endsWith("</feed>") == false) {
                    //logger.warn("The XML did not end with </feed>");
                    xml = xml + "</feed>";
                }

                InputStream inputStream = new ByteArrayInputStream(xml.getBytes(StandardCharsets.UTF_8));
                Document doc = docBuilder.parse(inputStream);
                return doc;

            } else {

                logger.error("getXMLDocument - status = " + status);

            }

        } catch (ClientProtocolException e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IOException e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (ParserConfigurationException e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (IllegalStateException e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (SAXException e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } catch (Exception e) {
            logger.error("getXMLDocument ERROR | SHORT ERROR MESSAGE: " + e.getMessage() + " FULL ERROR MESSAGE: " + e.toString());
        } finally {
            // When HttpClient instance is no longer needed,
            // shut down the connection manager to ensure
            // immediate deallocation of all system resources
            httpclient.getConnectionManager().shutdown();
        }

        logger.warn("Exiting from getXMLDocument but returning null");
        return null;
    }
}