ScrollViewer keeps snapping back to top

Question

So I have this user control that contains a ListView, wrapped by a ScrollViewer. I set up my layout in the DataTemplate and it's displaying fine, but when you try to scroll down, it just snaps back to the top; it won't hold its position.

Here is the xaml:

                            <Border BorderThickness="1" BorderBrush="Black" Grid.Column="0" Grid.Row="0" 
                        Grid.RowSpan="2" Height="80" Width="80" VerticalAlignment="Top">
                                <Rectangle>
                                    <Rectangle.Fill>
                                        <ImageBrush Stretch="Fill" ImageSource="{Binding ImageUri}"/>
                                    </Rectangle.Fill>
                                </Rectangle>
                            </Border>

                            <TextBlock Grid.Column="1" Grid.Row="0" Style="{StaticResource DefaultText}" Text="{Binding Name}" FontWeight="Bold" VerticalAlignment="Top" HorizontalAlignment="Left" />
                            <TextBlock Grid.Column="1" Grid.Row="1" Style="{StaticResource DefaultText}" Text="{Binding Description}" VerticalAlignment="Top" HorizontalAlignment="Left" />
                            <TextBlock Grid.Column="2" Grid.Row="0" Style="{StaticResource DefaultText}" Grid.RowSpan="2" Text="{Binding Cost}" Width="100" FontSize="40" HorizontalAlignment="Right" />

                        </Grid>
                    </Border>
            </DataTemplate>
        </ListView.ItemTemplate>
    </ListView>
</ScrollViewer>

I saw another post suggesting that the problem is that the ScrollViewer is larger than the grid inside the item template. First of all, that sounds silly, so if I'm just using scrollviewer wrong, please just let me know how to fix that. Secondly, when I take the advice I saw in the article (set the height of the inner grid to be larger than the height of the scrollviewer), what happens is the height of every list item ends up being the height of the grid.

Any suggestions?