How can I monitor memory and CPU usage by spark ap

问题:

After running my spark application, I want to monitor its memory and cpu usage to evaluate its performance but couldn't find any option. Is it possible to monitor it? How can I monitor memory and CPU usage by spark application?

回答1:

There are a few options:

1. Ganglia is one
2. If you're running on your own cluster, HDP or Cloudera both have real time CPU & memory consumption charts.
3. If you want specific JVM metrics, then I'd recommend FlameGraph, though it's not real time.
4. There's also Grafana, it's extremely powerful, you can track many metrics with it, and it's real time.