Implement UI virtualization in custom ItemsPanel

My goal is to display some layout like the official Android Imgur app where items have fixed width but variable height. I already have a custom panel with a GridView. It works. However, it breaks a couple of things:

I don't get UI virtualization. This is a big performance bottleneck when lots of images need to be rendered.
Since virtualization is gone, incremental loading isn't supported either.

I've searched around a lot, but I couldn't find any documentation on how to implement virtualization. So, I have a couple of questions:

Is it even possible to get virtualization with custom panel? How?
Is it possible to use one of the existing panels (VirtualizedStackPanel for example) to get the layout I need?

I don't think you can get virtualization on custom panels. The last time I needed custom, virtualized panel was in a prototype of what you see in the Windows Photos app, where I've done all virtualization from scratch. I see two options for you then:

Implement virtualization yourself - put a panel (Canvas works great, but you can use a custom panel too) in a ScrollViewer, create some containers to fill a few screens worth of items and as you scroll - rearrange the containers and their content to always have some items in the realization window. It's fun and you can get amazing performance, especially if you pre-calculate and cache the layout of items on a background thread. You might need to put some effort into handling focus, keyboard input (you should probably always keep the focused container realized regardless of window position) or possibly accessibility, but it's going to be worth it.
Use your custom panel in a regular ListView, but instead of altering the ListView.ItemsPanel - leave it as default (ItemsStackPanel) and put your custom layout panel inside of the item template and have tall, but distinct rows of these panels. This means that your layout won't be jagged everywhere - items will need to align with the ListViewItems somewhere, but it's somewhat simpler to implement the basics. I think that's how the layout in the Photos app works. You will need to put a lot of effort to override the way a ListView handles input and displays selection visuals, but it might work for you. Remember that in this case your ItemsSource will need to have groups of items to perform custom layout on in each ListViewItem.

I'd choose the first option, but it's up to you. It's some more work initially, but overall it might end up being easier to get right, gives you freedom to lay things out without any rows and it will teach you something useful. The second option will just mean you'll have to mask built-in behaviors that you don't need anyway. I think the first option also separates the view layer better from the view model one as there is no layout-dependent grouping required in the ItemsSource, unless you implement some sort of view-layer aggregator ItemsSource that will process the items from the view model.

Read more about some theory and practice of virtualization in this great article:

ListView basics and virtualization concepts