Giraph's estimated cluster heap 4096MB ask is

Question

I'm running Giraph using Hadoop 2.5.2 on a 5 node cluster. But when I try to run the SimpleShortestPathsComputation example, I get this error:

Exception in thread "main" java.lang.IllegalStateException: Giraph's estimated cluster heap 2000MB ask is greater than the current available cluster heap of 0MB. Aborting Job.

So far I've been unable to determine why Giraph thinks the cluster has a 0MB heap. I've set YARN_HEAPSIZE and HADOOP_HEAPSIZE in yarn-env.sh and hadoop-env.sh respectively, and in yarn-site.xml I have:

 <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8960</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
  </property>

Any idea what is going on with this?

Answer 1

This turned out to be an environmental issue. Basically, what happens is this: Giraph asks the YARN ResourceManager for a list of running nodes, and sums up the heap available across the listed nodes, then compares it against it's own estimate of how much heap it needs. In my case, a glitch in my setup was causing YARN to fail to list any nodes, and so Giraph thought that 0MB of heap was available to the cluster.

The underlying problem was two-fold:

1. I am running on EC2 and forgot to add port 8032 to the security group, so nodes could connect to the resource manager.
2. I forgot to set the yarn resourcemanager hostname property and so the nodes were trying to connect to 0.0.0.0 initially.

Once those two issue are resolved, this problem went away.