Is there a specific use of pdist function of scipy

Question

my question is about use of pdist function of scipy.spatial.distance. Although I have to calculate the hamming distances between a 1x64 vector with each and every one of other millions of 1x64 vectors that are stored in a 2D-array, I cannot do it with pdist. Because it returns hamming distances between any two vector inside the same 2D array. I wonder if there is any way to make it calculate hamming distances between a specific index' vector and all others each.

Here is my current code, I use 1000x64 for now because memory error shows up with big arrays.

import numpy as np
from scipy.spatial.distance import pdist


ph = np.load('little.npy')

print pdist(ph, 'hamming').shape

and the output is

(499500,)

little.npy has a 1000x64 array. For example, if I want only to see the hamming distances with 31. vector and all others. What should I do?

Answer 1

You can use cdist. For example,

In [101]: from scipy.spatial.distance import cdist

In [102]: x
Out[102]: 
array([[0, 1, 1, 1, 1, 0, 0, 0],
       [0, 0, 1, 0, 0, 0, 1, 0],
       [0, 0, 0, 1, 1, 1, 0, 0],
       [1, 0, 1, 1, 0, 1, 1, 0],
       [1, 0, 1, 1, 0, 1, 1, 1],
       [0, 1, 0, 1, 0, 0, 0, 1],
       [1, 0, 0, 0, 0, 1, 0, 0],
       [1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 0, 0, 1, 1, 1, 0],
       [1, 0, 0, 1, 1, 0, 0, 1]])

In [103]: index = 3

In [104]: cdist(x[index:index+1], x, 'hamming')
Out[104]: 
array([[ 0.625,  0.375,  0.5  ,  0.   ,  0.125,  0.75 ,  0.375,  0.375,
         0.5  ,  0.625]])

That gives the Hamming distance between the row at index 3 and all the other rows (including the row at index 3). The result is a 2D array, with a single row. You might want to immediately pull out that row so the result is 1D:

In [105]: cdist(x[index:index+1], x, 'hamming')[0]
Out[105]: 
array([ 0.625,  0.375,  0.5  ,  0.   ,  0.125,  0.75 ,  0.375,  0.375,
        0.5  ,  0.625])

I used x[index:index+1] instead of just x[index] so that input is a 2D array (with just a single row):

In [106]: x[index:index+1]
Out[106]: array([[1, 0, 1, 1, 0, 1, 1, 0]])

You'll get an error if you use x[index].