Retrieve SparkContext from SparkSession

问题:

I'm running a spark batch job and uses SparkSession as I need a lot of spark-sql features to process in each of my components. The SparkContext is initialized in my parent component and been passed to the child components as SparkSession.

In one of my child components, I wanted to add two more configurations to my SparkContext. Hence, I need to retrieve the SparkContext from the SparkSession, stop it and recreate the SparkSession with the additional configuration. To do so, how can I retrieve SparkContext from SparkSession?

回答1:

Just to post as an answer - the SparkContext can be accessed from SparkSession using spark.sparkContext (no parenthesis)

回答2:

The sparkContext field does not seem to be public anymore(I am using Spark 2.3.2), however, you can retreive it using a method of the same name:

spark.sparkContext()

This is applicable to Spark Java only.