How to refer to a Spark DataFrame by name in spark

Question

Say I ran the following code and I forgot to assign the Spark dataframe iris to a variable in R and I can't use .Last.value to assign because I had run some other code right after copying the data to Spark.

library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "local")
copy_to(sc, iris)
2+2 # ran some other code so can't use .Last.value

How do I assing the Spark dataframe "iris" to a variable in R called iris_tbl?

Answer 1

copy_to provides additional name argument By default it is set to:

deparse(substitute(df))

so in your case the name will be iris. If you want more predictable behavior you should set the name manually:

copy_to(sc, iris, name = "foo")

Then you can access it dplyr way with tbl:

dplyr::tbl(sc, "foo")

or via Spark session:

sc %>% spark_session() %>% invoke("table", "foo") %>% sdf_register()

All production ready reader methods (copy_to shouldn't be used as anything else than a testing and development tool) require name, so you can reference tables the same way

spark_read_csv(sc, "bar", path)
tbl(sc, "bar")