php/symfony/doctrine memory leak?

Question

I'm having problems with a batch insertion of objects into a database using symfony 1.4 and doctrine 1.2.

My model has a certain kind of object called "Sector", each of which has several objects of type "Cupo" (usually ranging from 50 up to 200000). These objects are pretty small; just a short identifier string and one or two integers. Whenever a group of Sectors are created by the user, I need to automatically add all these instances of "Cupo" to the database. In case anything goes wrong, I'm using a doctrine transaction to roll back everything. The problem is that I can only create around 2000 instances before php runs out of memory. It currently has a 128MB limit, which should be more than enough for handling objects that use less than 100 bytes. I've tried increasing the memory limit up to 512MB, but php still crashes and that doesn't solve the problem. Am I doing the batch insertion correctly or is there a better way?

Here's the error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /Users/yo/Sites/grifoo/lib/vendor/symfony/lib/log/sfVarLogger.class.php on line 170

And here's the code:

public function save($conn=null){

    $conn=$conn?$conn:Doctrine_Manager::connection();

    $conn->beginTransaction();


    try {
        $evento=$this->object;


        foreach($evento->getSectores() as $s){

            for($j=0;$j<$s->getCapacity();$j++){

                $cupo=new Cupo();
                $cupo->setActivo($s->getActivo());
                $cupo->setEventoId($s->getEventoId());
                $cupo->setNombre($j);
                $cupo->setSector($s);

                $cupo->save();

            }
        }

        $conn->commit();
        return;
    }
    catch (Exception $e) {
        $conn->rollback();
        throw $e;
    }

Once again, this code works fine for less than 1000 objects, but anything bigger than 1500 fails. Thanks for the help.

Answer 1

Doctrine leaks and there's not much you can do about it. Make sure you use $q->free() whenever applicable to minimize the effect. Doctrine is not meant for maintenance scripts. The only way to work around this problem is to break you script to parts which will perform part of the task. One way to do that is to add a start parameter to your script and after a certain amount of objects had been processed, the script redirects to itself with a higher start value. This works well for me although it makes writing maintenance scripts more cumbersome.

Answer 2

Try to break circular reference which usually cause memory leaks with

$cupo->save();

$cupo->free(); //this call

as described in Doctrine manual.

Answer 3

Periodically close and re-open the connection - not sure why but it seems PDO is retaining references.

Answer 4

I have just did "daemonized" script with symfony 1.4 and setting the following stopped the memory hogging:

sfConfig::set('sf_debug', false);

Answer 5

What is working for me is calling the free method like this:

$cupo->save();
$cupo->free(true); // free also the related components
unset($cupo);

Answer 6

For me , I've just initialized the task like that:

// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();
$config = ProjectConfiguration::getApplicationConfiguration('frontend', 'prod', true);
sfContext::createInstance($config);

(WITH PROD CONFIG)
and use free() after a save() on doctrine's object

the memory is stable at 25Mo

memory_get_usage=26.884071350098Mo

with php 5.3 on debian squeeze