Apache Storm : Metrics log file is empty

I am trying to follow the example here

https://www.endgame.com/blog/storm-metrics-how

here is my storm.yaml

storm.zookeeper.servers:
- localhost


supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704


nimbus.host: localhost

ui.port: 8080
ui.host: localhost

storm.log.dir: /path/to/storm/logdir

topology.max.spout.pending: 5000

I tried running the topology in local and cluster mode. the metrics.log file is created at the location /path/to/storm/logdir but the file is empty! am i missing some configuration?

The problem is with the current log4j2 setup of Metrics in Storm and the fix is a little involved.

Issues

Firstly let me list some bugs from Storm's Jira. Reading the issues and fixes sheds a good amount of light onto the issue at hand with metrics logging:

STORM-584 whole cluster is sharing the metrics log
STORM-1673 package rename of LoggingMetricsConsumermissed metrics logger setup
STORM-1767 naming logger using fully qualified class name creates problems

To summarise the situation above:

Defining metrics logger in cluster.xml is not right as worker loggers (where metrics are being logged from) are defined using worker.xml. So it's possible that a worker will not have access to the metrics logger definition. It also creates problems of file ownership - the first worker to log metrics would own the log file preventing other loggers in other workers from appending to metrics log file.
After the rename of the package from backtype.storm.metric to org.apache.storm.metric the package rename missed worker.xml file.
Finally the metrics logger is named org.apache.storm.metric.LoggingMetricsConsumer which means that when trying to create it using LoggerFactory.getLogger(LoggingMetricsConsumer.class) inside the LoggingMetricsConsumer class, the logger that is actually created is a logger attached to ROOT logger and using A1 appender instead of METRICS appender.

The fix

The actual fix for me involved 2 things (please note that first step is optional)

Optionally sorting out log4j2 setup in worker.xml and cluster.xml files.
Simply make sure your logging setup has the metrics logger defined in worker.xml like it is in this github commit for Storm project fixing the STORM-1673 issue (which has made it to 1.0.0 and 2.0.0). Conversely please also make sure that the metrics logger is no longer defined in the cluster.xml file anymore.
Renaming the metrics logger and forking LoggingMetricsConsumer class to point it at a newly renamed logger (unatached to root logger):

Firstly make sure your metrics logger in worker.xml file is no longer named org.apache.storm.metric.LoggingMetricsConsumer but say METRICS_LOGGER as per this snippet:
```
<Logger name="METRICS_LOGGER" level="info" additivity="false">
    <appender-ref ref="METRICS"/>
</Logger>
```
Secondly "fork" LoggingMetricsConsumer class from the same github commit as above.
Please also note that by fork I mean whatever it takes to bring its source code into your project in order to modify it. You can also define your own class implementing IMetricsConsumer as long as you then attach it to your topology. Once you have your own LoggingMetricsConsumer class you have to modify the way it creates its metrics logger to:

public static final Logger METRICS_LOG = LoggerFactory.getLogger("METRICS_LOGGER");

Last note if you are using version of Storm different than 1.0.0 or 2.0.0

Just make sure you look at the LoggingMetricsConsumer source code and the versions of worker.xml and cluster.xml from the git branch your version has been cut from. This area has changed a lot and each change is not backwards compatible (package rename, logger move).