Duplicate entries into Elastic Search while logsta

问题:

I have been trying to send logs from logstash to elasticsearch.Suppose I am running a logstash instance and while it is running,I make a change to the file which the logstash instance is monitoring,then all the logs which have been previously saved in the elasticsearch are saved again,hence duplicates are formed.

Also,when the logstash instance is closed and is restarted again,the logs gets duplicated in the elasticsearch.

How do I counter this problem? How to send only the newest added entry in the file from logstash to elasticsearch? My logstash instance command is the following: bin/logstash -f logstash-complex.conf

and the configuration file is this:

input
 {
  file
 {

     path => "/home/amith/Desktop/logstash-1.4.2/accesslog1"

  }
}

filter
 {
  if [path] =~ "access"
 {
    mutate 
{
 replace =>
 { "type" => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    host => localhost 
    index => feb9
  }
  stdout { codec => rubydebug }
}

回答1:

As you mentioned in your question.

when the logstash instance is closed and is restarted again,the logs gets duplicated in the elasticsearch.

So, it probably you have delete the .since_db. Please have a look at here. Try to specific the since_db and start_position. For example:

input
{
    file
    {
        path => "/home/amith/Desktop/logstash-1.4.2/accesslog1"
        start_position => "end"
        sincedb_path => /home/amith/Desktop/sincedb
    }
}

回答2:

I got the solution. I was opening the file,adding a record and saving it ,due to which logstash treated the same file as a different file each time I saved it as it registered different inode number for the same file.

The solution is to append a line to the file without opening the file but by running the following command.

echo "the string you want to add to the file" >> filename

回答3:

[ELK stack] I wanted some custom configs in

/etc/logstash/conf.d/vagrant.conf

so the first step was to make a backup: /etc/logstash/conf.d/vagrant.conf.bk This caused logstash to add 2 entries in elasticseach for each entry in <file>.log; the same if i had 3 files in /etc/logstash/conf.d/*.conf.* in ES i had 8 entries for each line in *.log