I am using RoR to develop an application and a gem called searchkick, this gem internally uses elasticsearch. Everything works fine but on the production, we faced a weird issue, that after some time the site goes down. The reason we discovered was the memory on the server was being overused. We deleted some elasticsearch log files of the previous week and found out that the memory use was reduced to 47% from 92%. we use rolled logging, and logs are backed up each day. Now, the problem that we are facing is, with only 1 log file of the previous day, the memory grows higher. The log files are taking up a lot of space, even the current one takes 4GB!!!! How can I prevent that?
The messages are almost are warn level.
[00:14:11,744][WARN ][cluster.action.shard ] [Abdul Alhazred] [?][0] sending failed shard for [?][0], node[V52W2IH5R3SwhZ0mTFjodg], [P], s[INITIALIZING], indexUUID [4fhSWoV8RbGLj5jo8PVoxQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[?][0] failed recovery]; nested: EngineCreationFailureException[[?][0] failed to create engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /usr/lib64/elasticsearch-1.1.0/data/elasticsearch/nodes/0/indices/?/0/index/write.lock]; ]]
Looking at some of the SO questions, I'm trying to increase the ulimit or create a new node, so that the problem is also solved and size reduces. My limits.conf has 65535 for hard and soft nofile. Also in sysctl.conf fs.file-max more that 100000. Is there any other step that I could take to reduce the file size, moreover I'm not able to get insight into elasticsearch config changes.
If anyone could help. thanks
