YARN shell command to get number of containers and

Question

hadoop job -list only seems to show mapreduce jobs, but my resource manager UI shows things like "Allocated CPU VCores" and "Allocated Memory MB" for all jobs running on YARN (including things like Spark and Tez).

How can I get these results via the command line instead of going to the UI?

Answer 1

YARN supports different types of applications. "MapReduce" is one type of the application supported by YARN. If you use hadoop job (which is deprecated, you should use mapred job instead) or mapred job, you can only manipulate MapReduce jobs.

To view the status of the different types of applications (mapreduce, spark etc.), you should use YARN CLI.

For e.g. "yarn application -list -appStates ALL", lists the status of all the applications. The output of this command contains a column "Application-Type" which indicates the type of the application (for e.g. MAPREDUCE for MapReduce application).

You can get the details about the application, application attempt, containers etc. The commands for 2.7.1 version of Hadoop are given here: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html.

But, these commands don't return the details like "Allocated CPU VCores" and "Allocated Memory MB", through CLI.

yarn application -status {Application ID} command returns "Aggregate Resource Allocation" in terms of "MB-seconds" and "vcore-seconds"

For e.g. -status for one of my applications returned:

Aggregate Resource Allocation : 12865641 MB-seconds, 1041 vcore-seconds