When I explicitly call rdd.cache, I can see from the spark console storage tab that only a fraction of the rdd is actually cached. My question is where are the remaining parts? How does Spark decide which part to leave in cache?
The same question applies to the initial raw data read in by sc.textFile(). I understand these rdd's are automatically cached, even though the spark console storage table does not display any information on their cache status. Do we know how much of those are cached vs. missing?
cache() is the same as persist(StorageLevel.MEMORY_ONLY), and your amount of data probably exceeds the available memory. Spark then evicts caches in a "least recently used" manner.
You can tweak the reserved memory for caching by setting configuration options. See the Spark Documentation for details and look out for: spark.driver.memory, spark.executor.memory, spark.storage.memoryFraction
Not an expert, but I do not think that textFile() automatically caches anything; the Spark Quick Start explicitly caches a text file RDD: sc.textFile(logFile, 2).cache()
