CollectionView Cell changing size after scrolling

I have a Collection View nested in a Collection View. What I have done to try and have the cell size change dynamically based on content is create an outlet for the height of the child collection and change its constant after getting the content size.

This is a screen show of my IB constraints, and I'm using the height one highlighted.

So i have the child collection view (aqua's color box) constrained to the cell of the parent on 4 sides with height constant.

In my custom cell class, which holds the child collection view, I am setting the height like:

    let contentHeight = innerCollectionView.collectionViewLayout.collectionViewContentSize.height
        innerCollectionHeight.constant = contentHeight

And then in my view controller class which holds the parent collection view in viewDidLoad i specify the estimated item size with:

    let flowLayout = outerCollectionView.collectionViewLayout as! UICollectionViewFlowLayout
        flowLayout.estimatedItemSize = CGSize(width: view.frame.width - 20, height: view.frame.height / 4)
        flowLayout.minimumLineSpacing = 10
        flowLayout.sectionInset = UIEdgeInsets(top: 0, left: 10, bottom: 10, right: 10)

The result is that on the initial load everything displays correctly, however after scrolling the top cell which has 2 rows of data (green number icons) adjusts itself to be the same as the others. You can see in the gif below that the child collection view (the aqua'ish coloured box) is still the same size, it is the parent cell (orange box) that has reduced in size.

I'm stumped here as to what to do.

the bit where it goes back to full size is just the gif reseting it doesn't actually do that.

inside cellForItemAt (where outerCell is the normally declared cell) redeclare delegate and then reload the child collection:

outerCell.setCollectionViewDataSourceDelegate(dataSourceDelegate: self, forItem: indexPath.item)
outerCell.innerCollectionView.reloadData()
outerCell.innerCollectionView.layoutIfNeeded()