How to use PHP to Submit a Form and Retrieve a Fil

Question

I need to download a CSV file from http://www.carriersoftwaredata.com/Login.aspx?file=FHWA to my local server on a daily basis. My plan is to implement a cron job running a php script to do this. However, the page requires a username and password be inputted before allowing the file to be downloaded. If I visit this page in my web browser, enter the username and password, and submit the form then my browser will show me a Download Dialog box and begin to download the file.

How can I use PHP to submit the form and download the provided file?

This is what I am doing currently to get the necessary $_POST information.

//url for fhwa db file
$fhwa_url = 'http://www.carriersoftwaredata.com/Login.aspx?file=FHWA';

//get the contents of the fhwa page
$fhwa_login = file_get_contents($fhwa_url);

//load contents of fhwa page into a DOMDocument object
$fhwa_dom = new DOMDocument;
if (!$fhwa_dom->loadhtml($fhwa_login))
{
   echo 'Could not Load html for FHWA Login page.';
}
else
{
   //create a post array to send back to the server - keys relate to the name of the input
   $fhwa_post_items = array(
      '__VIEWSTATE'=>'',
      'Email'=>'',
      'Password'=>'',
      '__EVENTVALIDATION'=>'',
   );

   //create an xpath object
   $xpath = new DOMXpath($fhwa_dom);

   //iterate through the form1 form and find all inputs
   foreach($xpath->query('//form[@name="form1"]//input') as $input)
   {
      //get name and value of input
      $input_name = $input->getAttribute('name');
      $input_value = $input->getAttribute('value');

      //check if input name matches a key in the post array
      if(array_key_exists($input_name, $fhwa_post_items))
      {
         //if the input name is Email or Password enter the defined email and password
         switch($input_name)
         {
            case 'Email':
               $input_value = $email;
               break;
            case 'Password':
               $input_value = $pass;
               break;
         }//switch

         //assign value to post array
         $fhwa_post[$input_name] = $input_value;
      }// if
   }// foreach
}// if

This is how I am trying to submit the form - but it doesn't seem to be working in the way that I need. I'd like what is returned by stream_get_contents to be the contents of the CSV file I want to download.

   //get the url data and open a connection to the page
   $url_data = parse_url($fhwa_url);
   $post_str = http_build_query($fhwa_post);

   //create socket
   $fp = @fsockopen($url_data['host'], 80, $errno, $errstr, 30);
   fputs($fp, "POST $fhwa_url HTTP/1.0\r\n");
   fputs($fp, "Host: {$url_data['host']}\r\n");
   fputs($fp, "User-Agent: Mozilla/4.5 [en]\r\n");
   fputs($fp, "Content-Type: application/x-www-form-urlencoded\r\n");
   fputs($fp, "Content-Length: ".strlen($post_str)."\r\n");
   fputs($fp, "\r\n");
   fputs($fp, $post_str."\r\n\r\n");
   echo stream_get_contents($fp);
   fclose($fp);

Any help is definitely appreciated.

Answer 1

The approach I would have to this is to use curl. You are able to pass HTTP Basic Auth with the request if that is what the server is loading. After that you should be able to take the data retrieved from the curl request and save that to your local machine with something along the php fwrite function.

Here is an example I have used. I stripped out my fancy logic, just down to the basics. It also doesn't have the code for Basic Auth, but you should be able to find that in Google easily. I think this may be quite easier unless there is something else you have to do I over looked.

// Download data
$ch = curl_init('http://www.server.com/file.txt');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$rawdata = curl_exec($ch);
curl_close ($ch);

// Save data
$fp = fopen('new/file/location.txt','w');
fwrite($fp, $rawdata);
fclose($fp);

Answer 2

Looks like you need to be using the cURL library: http://www.php.net/manual/en/book.curl.php

Here's an example from their website of using post data with curl (the POSTed form data is the array $data):

 <?php

/* http://localhost/upload.php:
print_r($_POST);
print_r($_FILES);
*/

$ch = curl_init();

$data = array('name' => 'Foo', 'file' => '@/home/user/test.png');

curl_setopt($ch, CURLOPT_URL, 'http://localhost/upload.php');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

curl_exec($ch);
?>

Answer 3

I found out that what I was originally doing was working but I simply wasn't sending enough of the post information. I wrote a function to help me easily do this in the future as well.

/**
 * @author: jeremysawesome - www.dnawebagency.com
 * @description: save_remote_file - Requires a url, a post array and a
 *    filename. This function will send the post_array to the remote server
 *    and retrieve the remote file. The retrieved remote file will be
 *    saved to the filename specified by the filename parameter.
 *    Note: This function should only be used when specific post information
 *       must be submitted to the remote server before the file download
 *       will begin.
 *       For Example: http://www.carriersoftwaredata.com/Login.aspx?file=FHWA
 * @param: post_url - the url to send a post request to
 * @param: post_array - the post arguments to send to the remote server.
 * @param: filename - the name to save the retrieved remote file as
**/
function save_remote_file($post_url, $post_array, $filename)
{
   //get the url data
   $url_data = parse_url($post_url);
   $post_str = http_build_query($post_array);

   //build the headers to send to the form
   $headers = "POST $post_url HTTP/1.0\r\n";
   $headers .= "Host: {$url_data['host']}\r\n";
   $headers .= "User-Agent: Mozilla/4.5 [en]\r\n";
   $headers .= "Content-Type: application/x-www-form-urlencoded\r\n";
   $headers .= "Content-Length: ".strlen($post_str)."\r\n";
   $headers .= "\r\n";
   $headers .= $post_str."\r\n\r\n";

   //create socket and download data
   $fp = @fsockopen($url_data['host'], 80, $errno, $errstr, 30);
   fputs($fp, $headers);
   $remote_file = stream_get_contents($fp);
   fclose($fp);

   //save data
   $saved_file = fopen($filename,'w') or die('Cannot Open File '. $filename);
   fwrite($saved_file, $remote_file);
   fclose($saved_file);
}// save_remote_file

For those that care - or need to do something similar in the future, here is the complete code (without the above function).

---

<?php
/**
 * @author: jeremysawesome - www.dnawebagency.com
 * @desciption: This file retrieves the database information from Carrier Software.
 *    This should be run with a cron in order to download the files. The file works
 *    by getting the contents of the Carrier Software login page for the database. An
 *    array of post values is created based on the contents of the login page and the
 *    defined username and password. A socket is opened and the post information is
 *    passed to the login page.
**/

//define username and pass
$user_info = array(
   'user'=>'[USERNAME]',
   'pass'=>'[PASSWORD]'
);

//url for fhwa db file
$fhwa_url = 'http://www.carriersoftwaredata.com/Login.aspx?file=FHWA';

//get the contents of the fhwa page
$fhwa_login = file_get_contents($fhwa_url);

//load contents of fhwa page into a DOMDocument object
$fhwa_dom = new DOMDocument;
if (!$fhwa_dom->loadhtml($fhwa_login))
{
   die('Could not Load html for FHWA Login page.');
}
else
{
   //create a post array to send back to the server - keys relate to the name of the input - this allows us to retrieve the randomly generated values of hidden inputs
   $fhwa_post_items = array(
      '__EVENTTARGET' => '',
      '__EVENTARGUMENT' => '',
      '__VIEWSTATE' => '',
      'Email' => '',
      'Password' => '',
      'btnSubmit' => 'Submit',
      '__EVENTVALIDATION' => '',
   );

   //create an xpath object
   $xpath = new DOMXpath($fhwa_dom);

   //iterate through the form1 form and find all inputs
   foreach($xpath->query('//form[@name="form1"]//input') as $input)
   {
      //get name and value of input
      $input_name = $input->getAttribute('name');
      $input_value = $input->getAttribute('value');

      //check if input name matches a key in the post array
      if(array_key_exists($input_name, $fhwa_post_items))
      {
         //if the input name is Email or Password enter the defined email and password
         switch($input_name)
         {
            case 'Email':
               $input_value = $user_info['user'];
               break;
            case 'Password':
               $input_value = $user_info['pass'];
               break;
         }//switch

         //assign value to post array
         $fhwa_post[$input_name] = $input_value;
      }// if
   }// foreach

   //save the file - function shown above
   save_remote_file($fhwa_url, $fhwa_post, "my_data_folder/my_fhwa-db.zip");
}// if