Yarn : Automatic clearing of filecache & usercache

We are running a spark streaming job with yarn as resource manager, noticing that these two directories are getting filled up on the data nodes and we are running out of space when we only run for couple of min's

/tmp/hadoop/data/nm-local-dir/filecache

these directories are not getting cleared automatically , from my research found that this property need's to be set, yarn.nodemanager.localizer.cache.cleanup.interval-ms

Even after setting this up ..it's not automatically clearing out any help will be greatly appreciated

<configuration>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.aux-services</name>
    ~       <value>mapreduce_shuffle</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.resourcemanager.hostname</name>
    ~       <value>hdfs-name-node</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.memory-mb</name>
    ~       <value>16384</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.cpu-vcores</name>
    ~       <value>6</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.scheduler.maximum-allocation-mb</name>
    ~       <value>16384</value>
    ~   </property>
         <property>
    ~       <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
    ~       <value>3000</value>
    ~   </property>
    ~
    ~   <!-- Needs to be explicitly set as part of a workaround for YARN-367.
    ~      | If changing this property, you must also change the
    ~      | hadoop.tmp.dir property in hdfs-site.xml. This location must always
    ~      | be a subdirectory of the location specified in hadoop.tmp.dir. This
    ~      | affects all versions of Yarn 2.0.0 through 2.7.3+. -->
    ~   <property>
    ~       <name>yarn.nodemanager.local-dirs</name>
    ~       <value>file:///tmp/hadoop/data/nm-local-dir</value>
    ~   </property>
    ~
    ~ </configuration>

回答1:

If the primary problem is that you are running out of space then try setting a lower value for the yarn property "yarn.nodemanager.localizer.cache.target-size-mb". By default this is 10240 MB(10GB).

As for automatic clean up not getting triggered, well it may be due to (or at-least related to) this unresolved bug reported on Yarn 2.7.1: https://issues.apache.org/jira/browse/YARN-4540

回答2:

Your cache cleanup interval is good, but as the local directory is in /tmp, it may be filling up very fast, usually /tmp will have less space. My recommendation is change your yarn.nodemanager.local-dirs to any storage disks like /u01

The recommended value for yarn.nodemanager.localizer.cache.cleanup.interval-ms is 600000 or 10 mins