where are my logstash metrics in graphite?

Question

This is probably a noobie question, but am having a hard time finding the answers, so I hope you guys can help me here.

I have a running logstash instance shipping logs from one server to another server which is running graphite.

Here is my output config

output {
    stdout { codec => rubydebug }
    graphite {
        host => "xxxxxxx.yyyy.amazonaws.com"
        port => 2003
        type => "logstash-metrics"
        metrics => ["logstash.%{remote_addr}", "logstash.%{status}"]
    }
}

I have checked the firewall is not blocking TCP 2003 on xxxxxxx.yyyy.amazonaws.com where has graphite running, however when I go to graphite's UI can not seem to find any of my metrics, I am wondering what could be the reason?

Thanks!

Answer 1

The following does not look like an answer but since your question is that of a debugging nature, this is the best form that i can come up with-

1. Make sure the graphite stack is working. The easiest way is to run this on the shell a few times and verify that the corresponding graph appears on graphite-

   echo "test.first 10 `date +%s`"| nc graphite.example.com 2003.

2. Since you do not seem to have statsd in the stack, you don't have to check if it is correctly relaying.

3. Now graphite logs whatever it receives. The default location is /opt/graphite/storage/log/carbon-cache/carbon-cache-a/.

listener.log- logs whenever NW connections are opened and closed.

06/12/2013 06:09:58 :: MetricLineReceiver connection with 127.0.0.1:59766 established
06/12/2013 06:10:00 :: MetricLineReceiver connection with 127.0.0.1:59766 closed cleanly

updates.log- logs metric updations.

06/12/2013 06:15:39 :: wrote 1 datapoints for stats.message.service.time_taken.std in 0.00017 seconds
06/12/2013 06:15:39 :: wrote 1 datapoints for exchange.message.job.service.time_taken.sum in 0.00016 seconds

creates.log- logs creation of new .wsp files for new metrics.

06/12/2013 06:17:31 :: new metric event.response.time_taken.sum_80 matched schema com
06/12/2013 06:17:31 :: new metric event.response.time_taken.sum_80 matched aggregation schema timers_fall_here
06/12/2013 06:17:31 :: creating database file /opt/graphite/storage/whisper/event/response/time_taken/sum_80.wsp (archive=[(300, 105120)] xff=0.0 agg=average)

Going through these you can find out weather the connection is not being created (NW issue) or if the wsp file creation isn't happening (file system permission issue). If sending metrics to graphite using nc works, then it is the logstash end that needs to be looked into.