Doctrine2 bulk import try to work with another ent

Question

I'm working on a members import batch (with insertions and updates) for a big project with a lot of entities such as Member, Client, Group, ....

After reading the chapter related to bulk imports in Doctrine doc, I've implemented this code :

$batchSize = 20;
$i         = 0;

foreach ($entities as $entity)
{
    $this->getEntityManager()->persist($entity);

    if (($i % $batchSize) === 0)
    {
        $this->getEntityManager()->flush();
        $this->getEntityManager()->clear();
    }
}

$this->getEntityManager()->flush();
$this->getEntityManager()->clear();

Now, when I want to bulk handle an array of Member entities, Doctrine try to insert null data into a completely other table related to the Group entity and an exception is thrown An exception occurred while executing 'INSERT INTO groups ...

There are not any relations between Member and Group ...

Any idea about this weird behavior ?

EDIT

Short mapping details :

/**
 * @ORM\Entity
 * @ORM\Table(name="members")
 */
class Member
{
    // some properties ...

    /**
     * @ORM\ManyToOne(targetEntity="Client", inversedBy="members", cascade={"persist", "merge"})
     * @ORM\JoinColumn(name="client_id", referencedColumnName="id", onDelete="CASCADE")
     */
    protected $client;

    /**
     * @return Client
     */
    public function getClient()
    {
        return $this->client;
    }

    /**
     * @param Client $client
     *
     * @return $this
     */
    public function setClient(Client $client)
    {
        $this->client = $client;

        return $this;
    }
}

/**
 * @ORM\Entity
 * @ORM\Table(name="clients")
 */
class Client
{
    /**
     * @ORM\OneToMany(targetEntity="Member", mappedBy="client", cascade={"persist", "remove", "merge"}, fetch="EXTRA_LAZY")
     */
    protected $members;

    /**
     * @ORM\ManyToOne(targetEntity="Group", inversedBy="clients", cascade={"persist", "merge"})
     * @ORM\JoinColumn(name="clients_id", referencedColumnName="id", onDelete="SET NULL")
     */
    protected $group;

    public function __construct()
    {
        $this->members = new ArrayCollection();
    }

    /**
     * @return ArrayCollection
     */
    public function getMembers()
    {
        return $this->members;
    }

    /**
     * @param $members
     *
     * @return $this
     */
    public function setMembers($members)
    {
        $this->members = new ArrayCollection();

        return $this->addMembers($members);
    }

    /**
     * @param $members
     *
     * @return $this
     */
    public function addMembers($members)
    {
        foreach ($members as $member)
        {
            $this->addMember($member);
        }

        return $this;
    }

    /**
     * @param Member $member
     *
     * @return $this
     */
    public function addMember(Member $member)
    {
        $this->members->add($member);
        $member->setClient($this);

        return $this;
    }

    /**
     * @param Member $member
     *
     * @return $this
     */
    public function removeMember(Member $member)
    {
        if ($this->members->contains($member))
        {
            $this->members->removeElement($member);
        }

        return $this;
    }

    /**
     * @param $members
     *
     * @return $this
     */
    public function removeMembers($members)
    {
        foreach ($members as $member)
        {
            $this->removeMember($member);
        }

        return $this;
    }

    /**
     * @param Group $group
     *
     * @return $this
     */
    public function setGroup(Group $group = null)
    {
        $this->group = $group;

        return $this;
    }

    /**
     * @return Group
     */
    public function getGroup()
    {
        return $this->group;
    }
}

/**
 * @ORM\Entity
 * @ORM\Table(name="groups")
 */
class Group
{
    /**
     * @ORM\OneToMany(targetEntity="Client", mappedBy="group")
     */
    protected $clients;

    public function __construct()
    {
        $this->clients = new ArrayCollection();
    }

    /**
     * @return ArrayCollection
     */
    public function getClients()
    {
        return $this->clients;
    }

    /**
     * @param $clients
     *
     * @return $this
     */
    public function setClients($clients)
    {
        $this->clients = new ArrayCollection();

        return $this->addClients($clients);
    }

    /**
     * @param $clients
     *
     * @return $this
     */
    public function addClients($clients)
    {
        foreach ($clients as $client)
        {
            $this->addClient($client);
        }

        return $this;
    }

    /**
     * @param Client $client
     *
     * @return $this
     */
    public function addClient(Client $client)
    {
        if (!$this->clients->contains($client))
        {
            $this->clients->add($client);
            $client->setGroup($this);
        }

        return $this;
    }

    /**
     * @param $clients
     *
     * @return $this
     */
    public function removeClients($clients)
    {
        foreach ($clients as $client)
        {
            $this->removeClient($client);
        }

        return $this;
    }

    /**
     * @param Client $client
     *
     * @return $this
     */
    public function removeClient(Client $client)
    {
        if ($this->clients->contains($client))
        {
            $this->clients->removeElement($client);
            $client->setGroup(null);
        }

        return $this;
    }
}

And the error is type of :

An exception occurred while executing 'INSERT INTO groups ... SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "label" violates not-null constraint DETAIL: Failing row contains (60, null, f, null, f, null, null).

EDIT2

This is the table creation description (using postgresql) :

CREATE TABLE groups (
    id integer NOT NULL,
    tempref character varying(255) DEFAULT NULL::character varying,
    prorated_basis boolean NOT NULL,
    fixed_price_amount double precision,
    is_indexed boolean,
    pricing_grid pricing[],
    label character varying(255) NOT NULL
);

CREATE SEQUENCE groups
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;

ALTER SEQUENCE groups_id_seq OWNED BY groups.id;

ALTER TABLE ONLY pricing_groups ALTER COLUMN id SET DEFAULT nextval('groups_id_seq'::regclass);

ALTER TABLE ONLY groups
    ADD CONSTRAINT groups_pkey PRIMARY KEY (id);

Answer 1

I can describe what is causing the error, but only guess why it is caused and give some hints on what to look for when debuging this.

As you described, you are updating members, that are part of a client, that in turn is part of a group. As you specified on the relations by cascade=persist, clients and groups are saved as well when persisting a member. That means, groups are either updated or created when inserting members. In your case, you are creating a new group by this mechanism. Yet this group does not have the label property set, resulting in a NULL value in the database, which is not allowed by the scheme.

As you said, this error is already occuring during the best batch. One of the first 20 members you update implicity creates a new group with no label. To find out which one it is I'd suggest using a debugger and inspecet each member before persistence to see what the group of this member is part of, and if it exists in the database. If it does not exist (by ID), you should investigate why this group does not the required label set.

If all groups actually do exist in the database already, things do get a bit more tricky and this depends on how the members you are updating are loaded. Are they fetched from the EntityManager (managed state) or are they loaded from some different source (e.g. serialized) and hence in a unmanaged state? If they are unmanaged, they will become manage upon peristence, and by specification of the relation cascade=merge, client, and group, will become managed as well. There is an important thing to know here though, merge will return a new (managed) entity which is then persisted (see the accepted answer here). As this is a new object, there might be the chance that this object is not fully initialized and can contain undefined values (which then would translate to NULL). So when loading the member data from a different source than the EntityManager, you might have to connect them with the EntityManager first to avoid this problem.

Debugging the last one is quite difficult and you'd need to step into the UnitOfWork->doPersist method to see how each individual entity is actual persisted.