I am trying to create an index on elasticsearch cloud using a json file. I have created the configuration as given below:
input {
file {
path => ["/root/leads.json"]
start_position => "beginning"
ignore_older => 0
}
}
output {
elasticsearch {
hosts => ["https://ac9xxxxxxxxxxxxxb.us-east-1.aws.found.io:9243"]
user => "elastic"
password => "xxxxxxxxxxxxxx"
}
}
I am able to run the logstash using the command:
sudo bin/logstash -f /etc/logstash/conf.d/logstash.conf
The logstash starts a pipeline, but I am not seeing any index getting created in elasticsearch:
[INFO ] 2018-11-14 09:16:01.821 [[main]>worker1] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_43b5fa3acfcfc04b3df80a7c15c8d991", :path=>["/root/leads.json"]}
[INFO ] 2018-11-14 09:16:01.852 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2fda8150 run>"}
[INFO ] 2018-11-14 09:16:01.944 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2018-11-14 09:16:01.996 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2018-11-14 09:16:02.522 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
You should change your configuration like this:
Remove
ignore_older => 0
since that will effectively ignore files that are older than 0 seconds :-) Addingsincedb_path
makes sure that you can run the pipeline several from the beginning of the file.