Elasticsearch converting a string to number

Question

I am new to Elasticsearch and am just starting up with ELK stack. I am collecting key value type logs in my Logstash and passing it to an index in Elasticsearch. I am using the kv filter plugin in Logstash. Due to this, all the fields are string type by default.

When I try to perform aggregation like avg or sum on a numeric field in Elasticsearch, I am getting an Exception: ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]

When I check the mappings in the index, all the fields except the timestamp ones are marked as string.

Please tell me how to overcome this issue as I have many numeric fields in my log events for aggregation.

Thanks,

Keerthana

Answer 1

In latest Logstash the syntax is as follows

filter {
  mutate {
    convert => { "fieldname" => "integer" }
  }
}

You can visit this link for more detail: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert

Answer 2

You could set explicit mappings for those fields (see e.g. Change default mapping of string to "not analyzed" in Elasticsearch for some guidance), but it's easier to just convert those fields to integers in Logstash using the mutate filter:

mutate {
    convert => ["name-of-field", "integer"]
}

Then Elasticsearch will do a better job at guessing the best data type for your field(s).

(See also Data type conversion using logstash grok.)