How to suppress the “Stage 2===>” from the output

问题:

I have dataframe and trying to get distinct count and able to get distinct count successfully but whenever scala program is executing i'm getting this message ([Stage 2:=============================> (1 + 1) / 2])how can i suppress particular this message in console.

val countID = dataDF.select(substring(col("dataDF"),5,7).distinct().count()

回答1:

You need to set spark.ui.showConsoleProgress to false

I found this in the comments of the ticket for the addition of the progress bar.

https://issues.apache.org/jira/browse/SPARK-4017

I haven't seen it in any of the documentation though, it really should be added.

回答2:

If you want to do it by code. Add the following when you are creating the SparkContext:

import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}

Logger.getRootLogger.setLevel(Level.ERROR)  // Disabling "INFO" level logs (these lines must be before to create the SparkContext)
val conf = new SparkConf().set("spark.ui.showConsoleProgress", "false").setAppName("myApp")  
val sc = new SparkContext(conf)

UPDATE CONTENT FOR SPARK2+:

Using SparkSession you can supress those messages adding the following line (.config("spark.ui.showConsoleProgress", "false")) to the declaration:

spark = SparkSession
      .builder
      .master("local[*]")
      .appName("myApp")
      .config("spark.ui.showConsoleProgress", "false")