Why does `substitute` work in multiple lines, but

Question

I was attempting to answer this nice question about creating a non-standard evaluating function for a data.table object, doing a grouped sum. Akrun came up with a lovely answer which I'll simplify here:

akrun <- function(data, var, group){
 var <- substitute(var)
 group <- substitute(group)
 data[, sum(eval(var)), by = group]
}

library(data.table)
mt = as.data.table(mtcars)
akrun(mt, cyl, mpg)
#    group    V1
# 1:     6 138.2
# 2:     4 293.3
# 3:     8 211.4

I was also working on an answer, and had close to the same answer, but with the substitutes inline with the rest. Mine results in an error:

gregor = function(data, var, group) {
  data[, sum(eval(substitute(var))), by = substitute(group)]
} 

gregor(mt, mpg, cyl)
# Error in `[.data.table`(data, , sum(eval(substitute(var))), by = substitute(group)) : 
#  'by' or 'keyby' must evaluate to vector or list of vectors 
#  (where 'list' includes data.table and data.frame which are lists, too) 

At its face, my function is a simple substitution of Akrun's. Why doesn't it work?

---

Note that both substitutions cause problems, as shown here:

gregor_1 = function(data, var, group) {
  var = substitute(var)
  data[,sum(eval(var)), 
       by = substitute(group)]
} 
gregor_1(mt, mpg, cyl)
# Same error as above


gregor_2 = function(data, var, group) {
  group = substitute(group)
  data[,sum(eval(substitute(var))), 
       by = group]
} 
gregor_2(mt, mpg, cyl)
# Error in eval(substitute(var)) : object 'mpg' not found 

Answer 1

data.table uses NSE because it needs to analyse/manipulate the by argument before choosing if it will evaluate it or not (if you give it a symbol for example it won't evaluate it).

A consequence is that if the argument needs to be evaluated it should be evaluated in the right environment and this is the function's responsibility. data.table evaluates its by argument in the data, not in the calling environment.

In most cases you don't see the issue as the symbol will be evaluated in the parent environment if not found, but substitute() is more sensitive.

See example below :

fun <- function(x){
  standard_eval(x)
  non_standard_eval_safe(x)
  non_standard_eval_not_safe(x)
}

standard_eval          <- function(expr) print(expr)

non_standard_eval_safe <- function(expr) {
  expr <- bquote(print(.(substitute(expr)))) # will be quote(print(x)) in our example
  eval.parent(expr)
}

non_standard_eval_not_safe <- function(expr) {
  expr <- bquote(print(.(substitute(expr))))  # will be quote(print(x)) in our example
  eval(expr)
}

standard_eval(1+1)          
#> [1] 2

non_standard_eval_safe(1+1)
#> [1] 2

non_standard_eval_not_safe(1+1)
#> [1] 2

fun(1+1)
#> [1] 2
#> [1] 2
#> Error in print(x): object 'x' not found

^{Created on 2020-02-20 by the reprex package (v0.3.0)}

Answer 2

In substitute's documentation you can read how it decides what to substitute, and the fact that, by default, it searches the environment where it is called. If you call substitute inside the data.table frame (i.e. inside []) it won't be able to find the symbols because they are not present inside the data.table evaluation environment, they are in the environment where [ was called.

You can "invert" the order in which the functions are called in order to get the behavior you want:

library(data.table)

foo <- function(dt, group, var) {
    eval(substitute(dt[, sum(var), by = group]))
}

foo(as.data.table(mtcars), cyl, mpg)
   cyl    V1
1:   6 138.2
2:   4 293.3
3:   8 211.4

Answer 3

It seems that substitute does not work within data table in the way one might expect from how it works in other contexts but you can use enexpr from the rlang package in place of substitute:

library(data.table)
library(rlang)

gregor_rlang = function(data, var, group) {
  data[, sum(eval(enexpr(var))), by = .(group = eval(enexpr(group)))]
} 

gregor_rlang(mt, mpg, cyl)
##    group    V1
## 1:     6 138.2
## 2:     4 293.3
## 3:     8 211.4

environments

The problem seems to be related to environments as this works where we have specifically given the environment substitute should use.

gregor_pf = function(data, val, group) {
  data[, sum(eval(substitute(val, parent.env(environment())))), 
    by = c(deparse(substitute(group)))]
} 
gregor_pf(mt, mpg, cyl)
##      cyl    V1
## 1:     6 138.2
## 2:     4 293.3
## 3:     8 211.4