how to suppress Hadoop logging message on the cons

Question

These are the Hadoop Logging Message I was trying to surpress

11/10/17 19:42:23 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
11/10/17 19:42:23 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
11/10/17 19:42:23 INFO mapred.MapTask: soft limit at 83886080
11/10/17 19:42:23 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
11/10/17 19:42:23 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

I suppose they are configured by logg 4j.properties under the conf directory under the hadoop installation directory. No matter how I remove the logger, comment out the logger, and even rename the log4j.properties. They are still being printed. Please advise.

Also I suspect these massive messages will impact the overall performance if the volume is big. Thanks,

Answer 1

You have to increase the log level to WARN, I assume it is currently on INFO. Maybe you have to recompile because the properties may also contained in jar.

Answer 2

I have found solution for this.All that it needs is changing the configuration file of mapreduce.

1. mapreduce.map.log.level can take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.

2. mapreduce.reduce.log.level can also take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.So its better to make sure that "mapreduce.job.log4j-properties-file" is not set.

We have to set following properties in the mapred-site.xml.

<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>


<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>

​Now i can see no log messages on the console.But it also has disadvantage as we cant figure out any error if it occurs while executing the mapreduce code as no log messages are visible.