dplyr function returning false NAs

Question

I have the same question as posted here, but the question remains unanswered and I also am having the same problem.

I have attached a sample of my data here.

The version of R that I am using is 3.4.2 and the version of dplyr is 0.7.4.

To get everyone up to speed... After importing the data, I do these edits:

#specify which species are predators (pp = 1) and prey (pp = 0)
d1 = d1 %>%
    group_by(sps) %>% #grouped by species
    mutate(pp=ifelse(sps %in% c("MUXX", "MUVI","MEME"), 1,0)) #mutate to specify predators as 1 and prey as 0

d1$datetime=strftime(paste(d1$date,d1$time),'%Y-%m-%d %H:%M',usetz=FALSE) #converting the date/time into a new format 

head(d1) #visualize the first few lines of the data

d2 = d1 %>% filter(km %in% c("80")) #restricting the observations to just one location (km 80)

Now for where the problems arise (the NAs):

d2 = d2 %>% mutate(prev = dplyr::lag(pp)) 
#when I look at the output I see the lag function isn't working (shown below)

> d2
# A tibble: 209 x 10
# Groups:   sps [10]
  ID       date    km culv.id   type    sps   time    pp         datetime  prev
<int>     <fctr> <dbl>  <fctr> <fctr> <fctr> <fctr> <dbl>            <chr> <dbl>
1     1 2012-06-19    80       A    DCC  MICRO   2:19     0 2012-06-19 02:19    NA
2     2 2012-06-21    80       A    DCC   MUXX  23:23     1 2012-06-21 23:23    NA
3     3 2012-07-15    80       A    DCC   MAMO  11:38     0 2012-07-15 11:38    NA
4     4 2012-07-20    80       A    DCC  MICRO  22:19     0 2012-07-20 22:19     0
5     5 2012-07-29    80       A    DCC  MICRO  23:03     0 2012-07-29 23:03     0
6     8 2012-08-07    80       A    DCC   PRLO   2:04     0 2012-08-07 02:04    NA
7     9 2012-08-08    80       A    DCC  MICRO  23:56     0 2012-08-08 23:56     0
8    10 2012-08-09    80       A    DCC   PRLO  23:06     0 2012-08-09 23:06     0
9    11 2012-08-13    80       A    DCC  MICRO   0:04     0 2012-08-13 00:04     0
10   12 2012-08-13    80       A    DCC  MICRO   0:46     0 2012-08-13 00:46     0

Might anyone have any suggests for why the lag function isn't working?

Answer 1

In one of your previous operations, you specified group_by(sps), that group will stay attached to your data frame until you ungroup(). Some row level operations won't be affected by the group, but aggregate functions, and functions that evaluate based on values from more than one row will.

d2 <- d2 %>% ungroup() %>% mutate(prev = dplyr::lag(pp))

Also, as to what I noticed:

1. I saw in your header # Groups: sps [10]
2. The first instance of each sps value is NA, but the second instance of each is correctly 0

As a final edit however, the first value of lag() will always be NA, because there is no previous value. This is also true with group_by(sps), however it means you'll have 10 NA values, one for the first instance of each factor level. If you want the lagged value within the group, then you should not ungroup() and the function is working properly creating those NAs. You may replace those NAs with 0, or another value if appropriate.