pandas - Selecting pair of consecutive rows matchi

问题:

I have a dataframe that looks like this

>>> a_df
    state
1    A
2    B
3    A
4    B
5    C

What I'd like to do, is to return all consecutive rows matching a certain sequence. For instance, if this sequence is ['A', 'B'], then the rows whose state is A followed immediately by a B should be returned. In the above example:

>>> cons_criteria(a_df, ['A', 'B'])
    state
1    A
2    B
3    A
4    B

Or if the chosen array is ['A', 'B', 'C'], then the output should be

>>> cons_criteria(a_df, ['A', 'B', 'C'])
    state
3    A
4    B
5    C

I decided to do this by storing the current state, as well as the next state:

>>> df2 = a_df.copy()
>>> df2['state_0'] = a_df['state']
>>> df2['state_1'] = a_df['state'].shift(-1)

Now, I can match with respect to state_0 and state_1. But this only returns the very first entry:

>>> df2[(df2['state_0'] == 'A') & (df2['state_1'] == 'B')]
    state
1    A
3    A

How should I fix the logic here so that all the consecutive rows are returned? Is there a better way to approach this in pandas?

回答1:

I'd use a function like this

def match_slc(s, seq):
    # get list, makes zip faster
    l = s.values.tolist()
    # count how many in sequence
    k = len(seq)
    # generate numpy array of rolling values
    a = np.array(list(zip(*[l[i:] for i in range(k)])))
    # slice an array from 0 to length of a - 1 with 
    # the truth values of wether all 3 in a sequence match
    p = np.arange(len(a))[(a == seq).all(1)]
    # p tracks the beginning of a match, get all subsequent
    # indices of the match as well.
    slc = np.unique(np.hstack([p + i for i in range(k)]))
    return s.iloc[slc]

---

demonstration

s = pd.Series(list('ABABC'))

print(match_slc(s, list('ABC')), '\n')
print(match_slc(s, list('AB')), '\n')

2    A
3    B
4    C
dtype: object 

0    A
1    B
2    A
3    B
dtype: object 

回答2:

Here's a solution that works for me - but only for numerical row indexes. I made your dataframe a little bit more interesting, now it has 2 ABC patterns:

a_df=pd.DataFrame(['A','B','A','B','C','D','A','A','B','C','E'],
                  columns=["state"])

And here's the match pattern:

pattern = ['A','B','C']

This expression finds a set of the starting rows of each pattern:

starts = set(a_df[a_df['state']          =='A'].index) & 
         set(a_df[a_df['state'].shift(-1)=='B'].index) & 
         set(a_df[a_df['state'].shift(-2)=='C'].index)
print(starts)
# {2, 7}

In general:

starts = set.intersection(
           *[set(a_df[a_df['state'].shift(-i)==value].index) 
             for i,value in enumerate(pattern)])

This expression converts the starting row numbers into 3-row ranges and selects the row ranges:

result = [a_df.ix[range(i, i+3)] for i in starts]
print(result)
# [  state
# 2     A
# 3     B
# 4     C,   state
# 7     A
# 8     B
# 9     C]

In general:

result = [a_df.ix[range(i, i+len(pattern))] for i in starts]