1. create_summary function

create_summary function fetches activity data from database and loop through activities using for each loop, and insert data into the below text file like below:

zfilename71801404123.txt

A|201309|R|C|2|014000956|014000956|2200|201211|M|3118.72|35215.12|1639.96|40749.29|46183.13|44653.83|1529.3|||423|9999|EVERGREEN IMPLEMENT INC 
A|201309|R|C|2|014000956|014000956|2201|201211|M|0|13.86|0|15.22|13.86|15.22|-1.36|||423|9999|EVERGREEN IMPLEMENT INC

2. insertdatafromfile function

insertdatafromfile function will read the contents of the same text file[zfilename71801404123.txt] and insert them into the summary table using the command LOAD DATA.

Code

function RebuildSummary() {
    $random = date('dmyhis');
    $zfilename = "zfilename".$random;
    create_summary($zfilename);
    insertdata($zfilename);
}

function create_summary($zfilename)
{
    $activities // data from DB
    $filepath = $_SERVER['DOCUMENT_ROOT']."\z".$zfilename.".txt";
    foreach ($activities as $activity) {
        $sql_summary = "SELECT A.AcctDb as AcctDb, '" . $default->DeftReportPeriod . "' as SumReportPer, '" . $default->DeftReportBase . "' as SumReportBase, '" . $default->DeftPeriodBasis . "' as SumPeriodBasis, '" . $default->DeftBasisAdj . "' as SumBasisAdj, '" . $AcctNo . "' as AcctNo,'" . $AcctTaxId . "' as AcctTaxId, '" . $RevLoc . "' as SumRevLoc, '" . $YTDStart . "' as SumYtdStart, '" . $CurrFreq . "' as SumCurrFreq, '" . $Curr . "' as SumCurrAmt, '" . $Ytd . "' as SumYtdAmt, '" . $Lastcurr . "' as SumLastCurr, '" . $LastYTD . "' as SumLastYtd, '" . $Last12 . "' as SumLast12, '" . $Prior12 . "' as SumPrior12, '" . $Last12diff . "' as SumLast12Diff, A.AcctDateOpen as SumDateOpen, A.AcctDateClosed as SumDateClosed, A.GroupCode as SumGroupCode, A.AcctHomeLoc as SumHomeLoc, A.AcctBusName as SumBusName, A.ClassCode as SumClassCode, '" . $Currdiff . "' asSumCurrDiff, '" . $Ytddiff . "' as SumYtdDiff, '" . $Mon['0'] . "' as SumMon01, '" . $Mon['1'] . "' as SumMon02, '" . $Mon['2'] . "' as SumMon03, '" . $Mon['3'] . "' as SumMon04, '" . $Mon['4'] . "' as SumMon05, '" . $Mon['5'] . "' as SumMon06, '" . $Mon['6'] . "' as SumMon07, '" . $Mon['7'] . "' as SumMon08, '" . $Mon['8'] . "' as SumMon09, '" . $Mon['9'] . "' as SumMon10, '" . $Mon['10'] . "' as SumMon11, '" . $Mon['11'] . "' as SumMon12,'" . $Amt['0'] . "' as SumAmt01, '" . $Amt['1'] . "' as SumAmt02, '" . $Amt['2'] . "' as SumAmt03,'" . $Amt['3'] . "' as SumAmt04, '" . $Amt['4'] . "' as SumAmt05, '" . $Amt['5'] . "' as SumAmt06, '" . $Amt['6'] . "' as SumAmt07, '" . $Amt['7'] . "' as SumAmt08, '" . $Amt['8'] . "' as SumAmt09, '" . $Amt['9'] . "' as SumAmt10, '" . $Amt['10'] . "' as SumAmt11, '" . $Amt['11'] . "' as SumAmt12 FROM accounts A WHERE A.AcctDb = '" . $AcctDb . "' and A.AcctTaxId='" . $AcctTaxId . "' ;";
        $exist_activity1 = $this->db->query($sql_summary);
        $activities1 = $exist_activity1->result_array();

        $flag_index = 0;
        foreach ($activities1[0] as $key => $value) {
            if ($flag_index == 0) {
            }
            $result .= $value . "|";
            $flag_index = 1;
        }

        $j++;
        $result = rtrim($result, "|");
        $handle = fopen($filepath, 'a') or die('Cannot open file:  ' . $filepath);
        fwrite($handle, $result);
        $new_data = "\n";
        fwrite($handle, $new_data);
        $result = "";
    }
}


function insertdatafromfile($zfilename) {
    ini_set('memory_limit', '-1');
    $ext = mysql_real_escape_string("\n");
    $filepath = "z".$zfilename.".txt";
    $sqlstatement1 = "LOAD DATA INFILE " . "'" . $filepath . "'" . " 
    INTO TABLE summary
    FIELDS TERMINATED BY '|'
    LINES TERMINATED BY " . "'" . $ext . "'" . " 
    (AcctDb, SumReportPer, SumReportBase, SumPeriodBasis, SumBasisAdj, AcctNo,AcctTaxId, SumRevLoc, SumYtdStart, SumCurrFreq, SumCurrAmt, SumYtdAmt, SumLastCurr, SumLastYtd, SumLast12, SumPrior12, SumLast12Diff, SumDateOpen, SumDateClosed, SumGroupCode, SumHomeLoc, SumBusName, SumClassCode, SumCurrDiff, SumYtdDiff, SumMon01, SumMon02, SumMon03, SumMon04, SumMon05, SumMon06, SumMon07, SumMon08, SumMon09, SumMon10, SumMon11, SumMon12, SumAmt01, SumAmt02, SumAmt03, SumAmt04, SumAmt05, SumAmt06, SumAmt07, SumAmt08, SumAmt09, SumAmt10, SumAmt11, SumAmt12)
";
    mysql_query($sqlstatement1);
    $filepath = $_SERVER['DOCUMENT_ROOT']."\z".$zfilename.".txt";
    unlink($filepath);
}

System Configuration:

Processor: Intel(R) Xeon(TM) CPU 2.80Ghz, 2.79 Ghz (2 Processors)
Installed memory(RAM) : 6.00 GB
System Type: 64 bit Operating System
Server: Windows IIS 7

PHPINFO

max_input_time: 60000
max_file_uploads: 2048M
memory_limit: 20000M
post_max_size: 20000M
upload_max_filesize: 15000M

My Question:

I am getting the below error while calling the RebuildSummary function.

[17-Apr-2014 03:54:42 America/Los_Angeles] PHP Fatal error:  Out of memory (allocated 1517289472) (tried to allocate 64 bytes) in C:\HostingSpaces\wwwroot\system\database\drivers\mysql\mysql_result.php on line 162

I have enough memory on server side, still how the system display this error "Out of memory"

回答1:

1517289472 bytes ~= 1.4G so you're hitting a barrier around there

The first thing you should look at is you're setting memory_limit: 20000M which is 20G. You've only 6G on your system. PHP may be confused and defaulting to a lower limit or there may be a hard limit for your version of php running on windows 7.

OR you could be running 32 bit Apache/PHP processes, which have hard limits around 2G see this: Upper memory limit for PHP/Apache

回答2:

if you're going through a tonne of data, do not use fetch_all() which is what I'm assuming you are using within your database class.

you should fetch each record individually, e.g.:

while ($row = mysqli_fetch_array($link)) {
    // insert row into file
}

here is a simple benchmark someone did before to illustrate the difference in memory usage: https://stackoverflow.com/a/2771563/520857

spoiler: 100Mb vs 400bytes

回答3:

Doesn't matter how much ram you allow for PHP. If it's a 32bit-compile, it could never use more than 4gig utter max, and practical ~3gig.

As for your code, you're sucking in your ENTIRE database result into ram, then building a string (essentially doubling memory requirements), then dumping that string out to file. Since you're obviously running out of memory, you'll have to stream the string, e.g.

$fh = fopen(...);
$separator = '';
while($row = fetch_from_db()) {
     fwrite($fh, $sep . $row[0]);
     $separator = '|';
}

This is somewhat inefficient, but should be enough to give you the general idea. A more efficient solution would be to build a LIMITED length string and write it out periodically. e.g. when your concatentated string reaches (say) 1 meg in size, you write it out and then reset the string to empty.