show graphite invalid lines

Question

I see lots of lines like this in my graphite logs:

01/10/2014 21:07:12 :: [listener] invalid line received from client HOST:PORT, ignoring

It would greatly help if I could see the invalid line. Some documentation and tutorials suggest graphite would print the offending line directly after the invalid warning, but for me it doesn't. How can I enable this property?

Thanks.

Answer 1

So my attempt to troubleshoot this was a total hack but it worked for me.

Steps

• Edit protocol.py (/opt/graphite/lib/carbon/protocols.py on line 75 and add an additional log line

Before:

 class MetricLineReceiver(MetricReceiver, LineOnlyReceiver):
    delimiter = '\n'

  def lineReceived(self, line):
    try:
      metric, value, timestamp = line.strip().split()
      datapoint = (float(timestamp), float(value))
    except:
      log.listener('invalid line received from client %s, ignoring' % self.peerName )
      return

    self.metricReceived(metric, datapoint)

After:

 class MetricLineReceiver(MetricReceiver, LineOnlyReceiver):
    delimiter = '\n'

  def lineReceived(self, line):
    try:
      metric, value, timestamp = line.strip().split()
      datapoint = (float(timestamp), float(value))
    except:
      log.listener('invalid line received from client %s, ignoring' % self.peerName )
      log.listener('invalid line - [ %s ]' % line)
      return

    self.metricReceived(metric, datapoint)

• restart daemon in debug mode

  /usr/bin/python /opt/graphite/bin/carbon-cache.py --pid /opt/graphite/storage/carbon-cache-a.pid --debug start

• find problem metric and fix
• revert change to protocol.py
• restart carbon-cache as daemon

By doing this I was able to see exactly which metric was causing me grief and address it.

Hope that helps!