SplunkUF is not forwarding logs from the Kubernate

Question

I deployed Splunk Forwarder in my kubernetes cluster using this blog.

http://jasonpoon.ca/2017/04/03/kubernetes-logging-with-splunk/

I have 4 files at

/opt/splunk/etc/system/local

inputs.conf

server.conf

limits.conf

outputs.conf

my inputs.conf looks like this.

[default]
host = testtest

[monitor:///usr/local/tomcat/logs]
whitelist=test.log|.log_WHITELIST_|test
index= abc
sourcetype=log4j
[splunktcp://9997]
compressed = false

I am not able to get the log in Splunk UI. when I am deploying it as side car. but If I use same docker image locally but putting dummy logs file. i can see the logs. Then why it is not working with kubernates?

I have checked the splunkd.log as well.

/opt/splunk/var/log/splunk # tail splunkd.log

12-07-2018 10:43:38.793 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
12-07-2018 10:43:38.793 +0000 INFO  TailingProcessor - Adding watch on path: /usr/local/tomcat/logs.
12-07-2018 10:43:38.795 +0000 INFO  loader - Limiting REST HTTP server to 21845 sockets
12-07-2018 10:43:38.795 +0000 INFO  loader - Limiting REST HTTP server to 657 threads
12-07-2018 10:43:38.798 +0000 INFO  TailReader - Registering metrics callback for: batchreader0
12-07-2018 10:43:38.798 +0000 INFO  TailReader - Starting batchreader0 thread
12-07-2018 10:43:38.798 +0000 INFO  TailReader - Registering metrics callback for: tailreader0
12-07-2018 10:43:38.798 +0000 INFO  TailReader - Starting tailreader0 thread
12-07-2018 10:43:38.850 +0000 INFO  TcpOutputProc - Connected to idx=52.204.198.184:9997 using ACK.
12-07-2018 10:44:08.358 +0000 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.

Any suggestion how to fix this. Stuck here for days. Do I have to open any port under kubernates? although I can ping splunk server from the splunk container.

Answer 1

I would highly suggest looking for the alternatives for forwarding Kubernetes logs to Splunk:

• Open Source Free version of Splunk Connect for Kubernetes - this is a recommended by Splunk option for forwarding logs
• Monitoring Kubernetes with Collectord by Outcold Solutions (I represent this company) - this is a proprietary software, takes 10 minutes to set up to forward logs and collect metrics.