Google Apps Script login to website with HTTP requ

Question

I have a spreadsheet on my Google Drive and I want to download a CSV from another website and put it into my spreadsheet. The problem is that I have to login to the website first, so I need to use some HTTP request to do that.

I have found this site and this. If either of these sites has the answer on it, then I clearly don't understand them enough to figure it out. Could someone help me figure this out? I feel that the second site is especially close to what I need, but I don't understand what it is doing.

To clarify again, I want to login with an HTTP request and then make a call to the same website with a different URL that is the call to get the CSV file.

Answer 1

I have done a lot of this in the past month so I should be able to help you, we are trying to emulate the browsers behaviour here so first you need to use chrome's developer tools(or something similar) and note down the exact things the browser does like the form values posted, the url that is called and so on. The following example shows the general techinique to be used:

The first step is to login to the website and get the session cookie:

  var payload =
   {
     "user_session[email]" : "username",
     "user_session[password]" : "password",
   };// The actual values of the post variables (like user_session[email]) depends on the site so u need to get it either from the html of the login page or using the developer tools I mentioned.
  var options =
   {
     "method" : "post",
     "payload" : payload,
     "followRedirects" : false
   };
  var login = UrlFetchApp.fetch("https://www.website.com/login" , options);
  var sessionDetails = login.getAllHeaders()['Set-Cookie'];

We have logged into the website (In order to confirm just log the sessionDetails and match it with the cookies set by chrome). The next step is purely dependent on the website so I will give u a general example

var downloadPayload = 
      {
        "__EVENTTARGET" : 'ctl00$ActionsPlaceHolder$exportDownloadLink1', 
      };// This is just an example it may or may not be needed, if needed u need to trace the values from the developer tools.
var downloadCsv = UrlFetchApp.fetch("https://www.website.com/", 
                                  {"headers" : {"Cookie" : sessionDetails},
                                   "method" : "post",
                                   "payload" : downloadPayload,
                                  });
Logger.log(downloadCsv.getContentText())

The file should now be logged, you can then parse the csv using hte GAS inbuilt function and dump the data in the spreadsheet.

A few points to note:

• I have assumed that all form post values are static and can be hardcoded, in case this is not true then let me know I will give you a function that can extract values from the html.
• Some websites require the browser to send a token value(the value will be present in the html) along with the credentials. In this case you need to extract the values and then post it.