Qt QGraphicsView - layout not shrinking & sizeHint

Question

I have the following, quite simple setup:

In a QWidget that is displayed as a window (no parent), there is a single QVBoxLayout.
In that QVBoxLayout, there is a single custom QGraphicsView.
The size policy of that custom QGraphicsView is set to Preferred/Preferred and setHeightForWidth set to true (and overwritten in the custom class) - I want to preserve aspect ratio.

The whole constructor of the widget here:

graphicsView = new CustomGraphicsView();
QVBoxLayout *layout = new QVBoxLayout();
layout->setMargin(0);
QSizePolicy sp(QSizePolicy::Preferred, QSizePolicy::Preferred);
sp.setHeightForWidth(true);
graphicsView->setSizePolicy(sp);
layout->addWidget(graphicsView);
setLayout(layout);

The setup works, and the aspect ratio is maintained when dragging the width of the window bigger (the height grows with it).
But as soon as I drag the window width smaller, the aspect ratio of the graphics view is maintained, but the height of the window itself won't shrink. The result being a small graphics view with lots of space above and below it that shouldn't be there.

Investigating it further I was trying to find out where things break, so I overloaded all the sizeHint(), minimumSize(), minimumHeight(), etc. functions of my custom graphics view.
Just to discover that not a single one of them is ever being called while I am manually resizing the window. The only thing called as expected is heightForWidth - which returns my calculated value - but its return value is not applied when the window is shrinking.

So, not only do I not know why the layout won't shrink on its own despite having the Preferred vertical policy (which explicity says "The widget can be expanded, but there is no advantage to it being larger than sizeHint()").
I also don't know how the layout gets the size from the widget to begin with. I assumed sizeHint() since all of the documentation permanently refers to it, but that is obviously wrong in this case.

What I already tried is to set the vertical policy to every possible value. None of them would cause the window to grow and shrink as it is supposed to.

My current workaround:
I have added the resizeEvent(...) function to the widget and inside that, I manually resize() the whole widget if its height exceeds the value returned by the heightForWidth() function of the custom graphics widget.
Okay as far as workarounds go, but it leads to heavy flickering (as usual when resizing inside a resizeEvent).

Any ideas on either problem?