`gather` can't handle rownames

2019-08-22 21:09发布

问题:

allcsvs = list.files(pattern = "*.csv$", recursive = TRUE)

library(tidyverse)

##LOOP to redact the snow data csvs##
for(x in 1:length(allcsvs)) {
  df = read.csv(allcsvs[x], check.names = FALSE)

  newdf = df %>%
    gather(COL_DATE, SNOW_DEPTH, -PT_ID, -DATE) %>%
    mutate( 
      DATE = as.Date(DATE,format = "%m/%d/%Y"),
      COL_DATE = as.Date(COL_DATE, format = "%Y.%m.%d")
    ) %>%
    filter(DATE == COL_DATE) %>%
    select(-COL_DATE)

  ####TURN DATES UNAMBIGUOUS HERE####
  df$DATE = lubridate::mdy(df$DATE)
  finaldf = merge(newdf, df, all.y = TRUE)
  write.csv(finaldf, allcsvs[x])

  df = read.csv(allcsvs[x])
  newdf = df[, -grep("X20", colnames(df))]
  write.csv(newdf, allcsvs[x])
}

I am using the code above to populate a new column row-by-row using values from different existing columns, using date as selection criteria. If I manually open each .csv in excel and delete the first column, this code works great. However, if I run it on the .csvs "as is"

I get the following message:

Error: Column 1 must be named

So far I've tried putting -rownames within the parenthesis of gather, I've tried putting remove_rownames %>% below newdf = df %>%, but nothing seems to work. I tried reading the csv without the first column [,-1] or deleting the first column in R df[,1]<-NULL but for some reason when I do that my code returns an empty table instead of what I want it to. In other words, I can delete the rownames in Excel and it works great, if I delete them in R something funky happens.

Here is some sample data: https://drive.google.com/file/d/1RiMrx4wOpUdJkN4il6IopciSF6pKeNLr/view?usp=sharing

回答1:

You can consider to import them with readr::read_csv.

An easy solution with tidyverse:

allcsvs %>%
            map(read_csv) %>% 
            reduce(bind_rows) %>%
            gather(COL_DATE, SNOW_DEPTH, -PT_ID, -DATE) %>% 
            mutate( 
                            DATE = as.Date(DATE,format = "%m/%d/%Y"),
                            COL_DATE = as.Date(COL_DATE, format = "%Y.%m.%d")
            ) %>%
            filter(DATE == COL_DATE) %>%
            select(-COL_DATE)

With utils::read.csv, you are importing strings are factors. as.Date(DATE,format = "%m/%d/%Y") evaluates NA.

Update

Above solution returns one single dataframe. To write the each data file separately with the for loop:

for(x in 1:length(allcsvs)) {
  read_csv(allcsvs[x]) %>% 
    gather(COL_DATE, SNOW_DEPTH, -PT_ID, -DATE) %>% 
    mutate( 
      COL_DATE = as.Date(COL_DATE, format = "%Y.%m.%d")
    ) %>%
    filter(DATE == COL_DATE) %>%
    select(-COL_DATE) %>%
    write_csv(paste('tidy', allcsvs[x], sep = '_'))
}

Comparison

  • purrr:map and purrr:reduce can be used instead of for loop in some cases. Those functions take another functions as arguments.
  • readr::read_csv is typically 10x faster than base R equivalents. (More info: http://r4ds.had.co.nz/data-import.html). Also it can handle CSV files better.