I'm pretty new to YARN. I ran my oozie jobs which creates logs.
I can see yarn logs with
yarn logs -applicationId application_123456789_12345678
I want to know how I can delete those logs? Can I just delete the file to remove the logs?
I'm pretty new to YARN. I ran my oozie jobs which creates logs.
I can see yarn logs with
yarn logs -applicationId application_123456789_12345678
I want to know how I can delete those logs? Can I just delete the file to remove the logs?
There no yarn commands to delete yarn logs from CLI.
You can delete by using Linux
rm
by going to the yarn log directoryyarn.nodemanager.log-dirs
/application_${appid}
. Individual containers log directories will be below this, in directories namedcontainer_{$contid}
. Each container directory will contain the filesstderr
,stdin
, andsyslog
generated by that container.Once the job has completed the NodeManager will keep the log for each container for
${yarn.nodemanager.log.retain-seconds}
which is 10800 seconds by default ( 3 hours ) and delete them once they have expired. But if${yarn.log-aggregation-enable}
is enabled then the NodeManager will immediately concatenate all of the containers logs into one file and upload them into HDFS in${yarn.nodemanager.remote-app-log-dir}/${user.name}/logs/<application ID>
and delete them from the local userlogs directoryDocumentation indicates
yarn.nodemanager.delete.debug-delay-sec
:Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the
yarn.nodemanager.log-dirs
property (see also below).You must configure
yarn-default.xml
:Maybe retain seconds can be useful: