Filtering and comparing dates with Pandas

Question

I would like to know how to filter different dates at all the different time levels, i.e. find dates by year, month, day, hour, minute and/or day. For example, how do I find all dates that happened in 2014 or 2014 in the month of January or only 2nd January 2014 or ...down to the second?

So I have my date and time dataframe generated from pd.to_datetime

df
    timeStamp
0   2014-01-02 21:03:04
1   2014-02-02 21:03:05
2   2016-02-04 18:03:10

So if I filter by the year 2014 then I would have as output:

    timeStamp
0   2014-01-02 21:03:04
1   2014-02-02 21:03:05

Or as a different example I want to know the dates that happened in 2014 and at the 2nd of each month. This would also result in:

    timeStamp
0   2014-01-02 21:03:04
1   2014-02-02 21:03:05

But if I asked for a date that happened on the 2nd of January 2014

    timeStamp
0   2014-01-02 21:03:04

How can I achieve this at all the different levels?

Also how do you compare dates at these different levels to create an array of boolean indices?

Answer 1

You can filter your dataframe via boolean indexing like so:

df.loc[df['timeStamp'].dt.year == 2014]
df.loc[df['timeStamp'].dt.month == 5]
df.loc[df['timeStamp'].dt.second == 4]
df.loc[df['timeStamp'] == '2014-01-02']
df.loc[pd.to_datetime(df['timeStamp'].dt.date) == '2014-01-02']

... and so on and so forth.

Answer 2

I would just create a string series, then use str.contains() with wildcards. That will give you whatever granularity you're looking for.

s = df['timeStamp'].map(lambda x: x.strftime('%Y-%m-%d %H:%M:%S'))

print(df[s.str.contains('2014-..-.. ..:..:..')])
print(df[s.str.contains('2014-..-02 ..:..:..')])
print(df[s.str.contains('....-02-.. ..:..:..')])
print(df[s.str.contains('....-..-.. 18:03:10')])

Output:

        timeStamp
0 2014-01-02 21:03:04
1 2014-02-02 21:03:05
        timeStamp
0 2014-01-02 21:03:04
1 2014-02-02 21:03:05
        timeStamp
1 2014-02-02 21:03:05
2 2016-02-04 18:03:10
        timeStamp
2 2016-02-04 18:03:10

I think this also solves your question about boolean indices:

print(s.str.contains('....-..-.. 18:03:10'))

Output:

0    False
1    False
2     True
Name: timeStamp, dtype: bool

Answer 3

If you set timestamp as index and dtype as datetime to get a DateTimeIndex, then you can use the following Partial String Indexing syntax:

df['2014'] # gets all 2014
df['2014-01'] # gets all Jan 2014
df['01-02-2014'] # gets all Jan 2, 2014