NSCollectionView deselect multiple selection

Question

So I've been investigating on this topic now for quite a bit, but haven't found anything useful related to it. My problem is that I created an NSCollectionView that serves like an image handler, so where you can drop images, select one or even multiple images at a time and delete them if wanted. Selecting the pictures was rather easy and I achieved it with this bit of code:

func collectionView(_ collectionView: NSCollectionView, didSelectItemsAt indexPaths: Set<IndexPath>) {
        selectedCells = indexPaths
    }

If the selectedCells is changed it will update the collectionView with the respective highlight.

My problem is now deselecting them. If I click on another image or multiple images the previously selected image gets deselected, but if I click on an empty space or on another view nothing happens, here is the problem. How can I deselect the previously selected Image(s) if the view is being unfocused. I already tried the didDeselectItemsAt function but it never gets called since i'm using multiple selection. I Hope it didn't get too confusing :]

Any help would be greatly appreciated