How can I create single numpy feature array from t

Question

I am trying to create a training data file which is structured as follows:

[Rows = Samples, Columns = features]

So if I have 100 samples and 2 features the shape of my np.array would be (100,2) etc.

Data

The list bellow contains path-strings to the .nrrd 3D sample patch-data files which have been processed using method 01.

['/Users/FK/Documents/image/01/subject1F_200.nrrd',
'/Users/FK/Documents/image/01/subject2F_201.nrrd']

Lets call the directory dir_01. For testing purposes the following 3D patch can be used. It has the same shape as the .nrrd file when read:

subject1F_200_PP01 = np.random.rand(128,128, 128)
subject1F_201_PP01 = np.random.rand(128,128, 128)
# and so on...

The list bellow contains path-strings to the .nrrd 3D sample patch-data files which have been processed using method 02.

['/Users/FK/Documents/image/02/subject1F_200.nrrd',
'/Users/FK/Documents/image/02/subject2F_201.nrrd']

Lets call the directory dir_02. For testing purposes the following 3D patch can be used. It has the same shape as the .nrrd file when read:

subject1F_200_PP02 = np.random.rand(128,128, 128)
subject1F_201_PP02 = np.random.rand(128,128, 128)
# and so on...

Both the subjects are the same, but the patch data has been pre-processed differently.

Feature Functions

In order to calculate the features I need to use the following functions:

1. np.median (regular python function and returns a single value)
2. my_own_function1 (regular python function and returns a np.array)
3. my_own_function2 (I can only access it using a matlab engine and returns a np.array)

In this scenario my final numpy array should have a (2,251) shape. Since I have to samples (rows) and 251 features (columns) from my 3 functions.

Here is my code (credits to M.Fabré)

Read the patches

# Helps me read the files for features 1. and 2. Uses a python .nrrd reader
def read_patches_multi1(files_1):
    for file_1 in files_1:
        yield nrrd.read(str(file_1))

# Helps me read the files for features 3. Uses a matlab .nrrd reader
def read_patches_multi2(files_2):
    for file_2 in files_2:
        yield eng.nrrdread(str(file_2))

Calculate

def parse_patch_multi(patch1, patch2):

    # Structure for python .nrrd reader
    data_1 , option = patch1

    # Structure for matlab .nrrd reader
    data_2 = patch2

    # Uses itertools to combine single float32 value with np.array values
    return [i for i in itertools.chain(np.median(data_1), my_own_function1(data_1), my_own_function2(data_2))]

Execution

# Directories
dir_01 = '/Users/FK/Documents/image/01/'
dir_02 = '/Users/FK/Documents/image/02/'

# Method 01 patch data
file_dir_1 = Path(dir_01)
files_1 = file_dir_1.glob('*.nrrd')
patches_1 = read_patches_multi1(files_1)

# Method 02 patch data
file_dir_2 = Path(dir_02)
files_2 = file_dir_2.glob('*.nrrd')
patches_2 = read_patches_multi2(files_2)

# I think the error lies here...
training_file_multi = np.array([parse_patch_multi(patch1,patch2) for (patch1, patch2) in (patches_1, patches_2)], dtype=np.float32)

I have tried multiple approaches but I am keep getting syntax error or the wrong structure. Or the following type error:

TypeError: unsupported Python data type: numpy.ndarray

Answer 1

I found a solution but it does not seem too elegant

I create two funcitons:

def parse_patch_multi1(patch1):

    # Structure for python .nrrd reader
    data_1 , option = patch1

    # Uses itertools to combine single float32 value with np.array values
    return [i for i in itertools.chain(np.median(data_1), 0) my_own_function1(data_1)]


def parse_patch_multi2(patch2):

    # Structure for python .nrrd reader
    data_2 = patch2

    # Uses itertools to combine single float32 value with np.array values
    return [i for i in itertools.chain(my_own_function2(data_2)]

Execution

# Directories
dir_01 = '/Users/FK/Documents/image/01/'
dir_02 = '/Users/FK/Documents/image/02/'

# Method 01 patch data
file_dir_1 = Path(dir_01)
files_1 = file_dir_1.glob('*.nrrd')
patches_1 = read_patches_multi1(files_1)

# Method 02 patch data
file_dir_2 = Path(dir_02)
files_2 = file_dir_2.glob('*.nrrd')
patches_2 = read_patches_multi2(files_2)

training_file_multi1 = np.array([parse_patch_multi1(patch1) for (patch1) in patches_1], dtype=np.float32)
training_file_multi2 = np.array([parse_patch_multi2(patch2) for (patch2) in patches_1], dtype=np.float32)

The trick

concatenate the two np.arrays along Axis 1

training_file_combined= np.concatenate((training_file_multi1, training_file_multi2), axis=1)

Shape of the matrix (2,252)