genfromtxt and numpy

Question

I have data in files such as "file.csv". I would like to read them with np.genfromtxt and do some statistics like average, variance etc. on some columns (X, Y, Z). However I want to make the statistics on for X > 1, Y > 3 Z > 2 etc. This is a simple example here.

This code produces almost correct results but it includes ALL Xs, Ys and Zs, I want to do the same but with the X,Y,Z conditions i specified above.

#file.csv
X,Y,Z
1,2,3
4,2,5
15,9,1
#

data = np.genfromtxt(file.csv, delimiter=',', dtype=float, unpack=True, skiprows = 0) 
X=data[0];Y=data[1];Z=data[2]
Mean = np.average(X)

--> Doing a great job getting the average. However, I want i to get average ONLY IF X > 1 (for example)... How do I make it do so?

Answer 1

In order to average over only some fields, you break down your averaging as follows:

1. Find the indexes (ind) of those elements that meet a certain criteria
2. Find the mean of the array indexed only with the the values in ind

The following code does exactly this:

indexes = np.where(X>1)[0] # We index with '0' here to get to the 1st element of the returned tuple
Mean = np.mean(X[indexes])

Answer 2

You could use so-called "fancy-indexing", X[X>1], to select the part of the array you want:

import numpy as np
X,Y,Z = np.genfromtxt('file.csv', delimiter=',', dtype=float, unpack=True, skiprows = 0)
print(X)
# [ nan   1.   4.  15.]
print(X[X>1])
# [  4.  15.]
print(np.average(X[X>1]))
# 9.5

To combine two masks (boolean arrays) with bit-wise logical-and, use the & operator:

print(np.average(X[(X>1)&(X<10)]))
# 4.0