Get the number of fields on an index

Question

For optimization purposes, I am trying to cut down my total field count. However before I am going to do that I want to get an idea of how many fields I actually have. There doesn't seem to be any Information in the _stats endpoint and I can't quite figure out how the migration tool does its field count calculation.

Is there some way, either with an endpoint or by other means, to get the total field count of a specified index?

Answer 1

The first answer by Val solves the problem for me too. But I just wanted to list out some corner cases which can lead to misleading numbers.

1. The document has fields with the word "type" in them.

For example

 "content_type" : {
   "type" : "text",
     "fields" : {
       "keyword" : {
          "type" : "keyword",
       }
     }
   },

This will match grep type thrice while it should do that only twice i.e. it should not match "content_type". This scenario has an easy fix.

Instead of

curl -s -XGET localhost:9200/index/_mapping?pretty | grep type 

Use

curl -s -XGET localhost:9200/index/_mapping?pretty | grep '"type"'

to get an exact match of '"type"'

2. The document has a field of the exact name "type"

For example

"type" : {
  "type" : "text",
   "fields" : {
     "keyword" : {
       "type" : "keyword"
     }
   }
},

In this case also the match is thrice instead of twice. But using

curl -s -XGET localhost:9200/index/_mapping?pretty | grep '"type"'

is not going to cut it. We will have to skip fields with the "type" keyword as substring as well as an exact match. In this case we can add an additional filter like so:

curl -s -XGET localhost:9200/index/_mapping?pretty |\
grep '"type"' | grep -v "{"

In addition to the above 2 scenarios, if you are using the api programmatically to push numbers for tracking i.e. into something like AWS cloudwatch or Graphite, you can use the following code to call the API - get the data, and recursively search for the keyword "type" - while skipping any fuzzy matches and resolving deeper into fields with the exact name "type".

import sys
import json
import requests

# The following find function is a minor edit of the function posted here
# https://stackoverflow.com/questions/9807634/find-all-occurrences-of-a-key-in-nested-python-dictionaries-and-lists

def find(key, value):
  for k, v in value.iteritems():
    if k == key and not isinstance(v, dict) and not isinstance(v, list):
      yield v
    elif isinstance(v, dict):
      for result in find(key, v):
        yield result
    elif isinstance(v, list):
      for d in v:
        for result in find(key, d):
          yield result

def get_index_type_count(es_host):
  try:
    response = requests.get('https://%s/_mapping/' % es_host)
  except Exception as ex:
    print('Failed to get response - %s' % ex)
    sys.exit(1)

  indices_mapping_data = response.json()
  output = {}

  for index, mapping_data in indices_mapping_data.iteritems():
    output[index] = len(list(find('type', mapping_data)))

  return output

if __name__ == '__main__':
  print json.dumps(get_index_type_count(sys.argv[1]), indent=2)

The above code is also posted as a gist here - https://gist.github.com/saurabh-hirani/e8cbc96844307a41ff4bc8aa8ebd7459

Answer 2

You can try this:

curl -s -XGET "http://localhost:9200/index/_field_caps?fields=*" | jq '.fields|length'

Answer 3

To build a bit further upon what the other answer provided, you can get the mapping and then simply count the number of times the keyword type appears in the output, which gives the number of fields since each field needs a type:

curl -s -XGET localhost:9200/index/_mapping?pretty | grep type | wc -l

Answer 4

Just a quick way to get a relative estimate in Kibana without writing a script (I don't believe this is 100% precise, but it's an easy way to tell if your dynamic fields are blowing up to huge numbers for some reason).

Run this query in the Kibana dev tools

GET /index_name/_mapping

Inside the Kibana output, perform a search for all instances of "type" (including quotes). This will count the instances and get you the answer. (In this example, 804)

This can be helpful if you scratching your head as to why you're getting the [remote_transport_exception] error of

Limit of total fields [1000] in index [index_name] has been exceeded

Answer 5

A field can have more than one "type": e.g.

"datapath-id": {
    "fields": {
        "keyword": {
            "ignore_above": 256, 
            "type": "keyword"
        }
    }, 
    "type": "text"
}

We can ignore "type" within "fields" to get exact field count. One example is:

import json


def myprint(d, field_count):
    for k, v in d.iteritems():
        if isinstance(v, dict):
            if k != "fields":
                field_count = myprint(v, field_count)
        else:
            print "{0} : {1}".format(k, v)
            field_count += 1
    return field_count

with open("output/mappings.json") as f:
    d = json.load(f)
    final_field_count = myprint(d, field_count=0)
    print "field count", final_field_count

Answer 6

You can get that information with the _mapping endpoint of the index API, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

The get mapping API allows to retrieve mapping definitions for an index or index/type.

GET /twitter/_mapping/tweet

With curl: curl [elasticsearch adress]/[index]/_mapping?pretty