log rotation script for logstash to purge logs gre

Question

I'm trying to come up with the best way to purge the logs from a logstash server that are more than two weeks old.

For those that aren't aware, Logstash stores it's logs inside of Elasticsearch. We have a really great stable ELK stack (Elasticsearch/Logstash/Kibana) where I work.

The typical way of deleting a logstash index is with a curl command like this one:

#curl --user admin -XDELETE http://localhost:9200/logstash-2015.06.06
Enter host password for user 'admin':
{"acknowledged":true}

Now what I'm looking for is a programmatic way of changing the dates in the logstash index to automatically purge any index that's greater than two weeks old.

I'm thinking of using bash to get this done.

I'd appreciate any examples of how to do this or advice you may have!

Thanks

Thanks!! But do you think you can help me get this going using auth?

This is what I tried so far:

[root@logs:~] #curator --help | grep -i auth
  --http_auth TEXT   Use Basic Authentication ex: user:pass
[root@logs:~] #curator delete indices --older-than 14 --time-unit days --timestring %Y.%m.%d --regex '^logstash-' --http_auth admin:secretsauce
Error: no such option: --http_auth
[root@logs:~] #curator delete indices --older-than 14 --time-unit days --timestring %Y.%m.%d --regex '^logstash-' --http_auth admin:secretsauce
Error: no such option: --http_auth
[root@logs:~] #curator delete indices --http_auth admin:secretsauce --older-than 14 --time-unit days --timestring %Y.%m.%d --regex '^logstash-'
Error: no such option: --http_auth

Answer 1

The online documentation for Curator explains many of these details. The URL is handily provided at the top of the --help output:

$ curator --help
Usage: curator [OPTIONS] COMMAND [ARGS]...

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

There's an entire sub-section on flags. In the documentation for the --http_auth flag it says:

This flag must come before any command.

Answer 2

Use Curator. To delete indexes older than 14 days you can run this command:

curator delete indices --older-than 14 --time-unit days --timestring %Y.%m.%d --regex '^logstash-'

Answer 3

If curator doesn't work for you for one reason or another, here's a bash script you can run:

#!/bin/bash

: ${2?"Usage: $0 [number of days] [base url of elastic]"}

days=${1}
baseURL=${2}

curl "${baseURL}/_cat/indices?v&h=i" | grep logstash | sort --key=1 | awk -v n=${days} '{if(NR>n) print a[NR%n]; a[NR%n]=$0}' | awk -v baseURL="$baseURL" '{printf "curl -XDELETE '\''%s/%s'\''\n", baseURL, $1}' | while read x ; do eval $x ; done