How do I suppress the bloat of useless information

问题:

I'm working with PigLatin, using grunt, and every time I 'dump' stuffs, my console gets clobbered with blah blah, blah non-info, is there a way to surpress all that?

grunt> A = LOAD 'testingData' USING PigStorage(':'); dump A; 

2013-05-06 19:42:04,146 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-05-06 19:42:04,147 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false ...
...
--- another like 50 lines of useless context clobbering junk here... till --- ...
...
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

now my like 4 lines of info looking for:

(daemon,*,1,1,System Services,/var/root,/usr/bin/false)
(uucp,*,,,/var/spool/uucp,/usr/sbin/uucico)
(taskgated,*,13,13,Task Gate Daemon,/var/empty,/usr/bin/false)
(networkd,*,24,24,Network Services,/var/empty,/usr/bin/false)
(installassistant,*,25,25,/usr/bin/false)

grunt>                                                                                       

---> obviously if it errors, fine lotsa info helpful, but not when it basically works great.

回答1:

You need to set the log4j properties. For example:

$PIG_HOME/conf/pig.properties :
enable:
# log4jconf=./conf/log4j.properties

rename: log4j.properties.template -> log4j.properties

log4j.properties :
set info to error:
log4j.logger.org.apache.pig=info, A

You may also set the Hadoop related logging level as well:

log4j.logger.org.apache.hadoop = error, A

回答2:

An easy way to do this seems to be to redirect standard error as below.

But it will suppress all errors.

pig -x local 2> /dev/null

Also found that if you remove or rename your hadoop install directory to basically make it inaccessible to pig then all those INFO messages go away.
Changing logging levels in hadoop didn't help, just so that you know.

回答3:

When you start pig, pass it a log4j.properties file with pig -4 <filename>.

In my case there was a log4j.properties in the conf directory and setting the level of the logger named org.apache.pig to ERROR is sufficient to make the logger less verbose.

log4j.logger.org.apache.pig=ERROR, A

回答4:

pig has debug log level one need to set that in pig.properties file,

# Logging level. debug=OFF|ERROR|WARN|INFO|DEBUG (default: INFO)
#
# debug=INFO

The reason one get large logs on console, e.g. change it to ERROR