Batch insertion of data to MySQL database using ph

Question

I have a thousands of data parsed from huge XML to be inserted into database table using PHP and MySQL. My Problem is it takes too long to insert all the data into table. Is there a way that my data are split into smaller group so that the process of insertion is by group? How can set up a script that will process the data by 100 for example? Here's my code:

foreach($itemList as $key => $item){
     $download_records  = new DownloadRecords();
    //check first if the content exists
    if(!$download_records->selectRecordsFromCondition("WHERE Guid=".$guid."")){
         /* do an insert here */
    } else {
         /*do an update */
    }

}

*note: $itemList is around 62,000 and still growing.

Answer 1

Yes, just do what you'd expect to do.

You should not try to do bulk insertion from a web application if you think you might hit a timeout etc. Instead drop the file somewhere and have a daemon or cron etc, pick it up and run a batch job (If running from cron, be sure that only one instance runs at once).

Answer 2

Using a for loop?

But the quickest option to load data into MySQL is to use the LOAD DATA INFILE command, you can create the file to load via PHP and then feed it to MySQL via a different process (or as a final step in the original process).

If you cannot use a file, use the following syntax:

insert into table(col1, col2) VALUES (val1,val2), (val3,val4), (val5, val6)

so you reduce to total amount of sentences to run.

EDIT: Given your snippet, it seems you can benefit from the INSERT ... ON DUPLICATE KEY UPDATE syntax of MySQL, letting the database do the work and reducing the amount of queries. This assumes your table has a primary key or unique index.

To hit the DB every 100 rows you can do something like (PLEASE REVIEW IT AND FIX IT TO YOUR ENVIRONMENT)

$insertOrUpdateStatement1 = "INSERT INTO table (col1, col2) VALUES ";
$insertOrUpdateStatement2 = "ON DUPLICATE KEY UPDATE ";
$counter = 0;
$queries = array();

foreach($itemList as $key => $item){
    $val1 = escape($item->col1); //escape is a function that will make 
                                 //the input safe from SQL injection. 
                                 //Depends on how are you accessing the DB

    $val2 = escape($item->col2);

    $queries[] = $insertOrUpdateStatement1. 
    "('$val1','$val2')".$insertOrUpdateStatement2.
    "col1 = '$val1', col2 = '$val2'";

    $counter++;

    if ($counter % 100 == 0) {
        executeQueries($queries);
        $queries = array();
        $counter = 0;
    }
}

And executeQueries would grab the array and send a single multiple query:

function executeQueries($queries) {
   $data = "";
     foreach ($queries as $query) {
        $data.=$query.";\n";
    }
    executeQuery($data);
}

Answer 3

You should put it as said before in a temp directory with a cron job to process files, in order to avoid timeouts (or user loosing network).

Use only the web for uploads.

If you really want to import to DB on a web request you can either do a bulk insert or use at least a transaction which should be faster.

Then for limiting inserts by batches of 100 (commiting your trasnsaction if a counter is count%100==0) and repeat until all your rows were inserted.