Today I added some add-ons to my splunk and did some sysadmin on the server. Restarted and noted the splunkd was not running. Ahh, well, that's typical. Starting the splunk daemon is easy enough:
Start Splunk - from the people who made splunk.
There are two ways to start splunk, as you can read from above. One is to run the "splunk" process from your root shell after logging in. This will run splunk as root. The other is to use the nifty systemctl service script to daemonize the process.
Prior to today, I had the same problem and ran the splunk process as root. This was foolish. If you happen to have once started splunk as root, and then successfully started splunk as the "splunk" user, you will find that your splunk login page is empty. You get the background picture, but no input controls.
Damn. Google that, nada. Damn again.
Today, I learned alot more about selinux and permissions and labels, so I investigated the "web_service" log (/opt/splunk/var/log/splunk/web_service.log) and found:
IOError: [Errno 13] Permission denied: '/opt/splunk/var/run/splunk/session-d07528932b4314e72a9f5bccd0f85fb27e8f30bd.lock'
Start Splunk - from the people who made splunk.
There are two ways to start splunk, as you can read from above. One is to run the "splunk" process from your root shell after logging in. This will run splunk as root. The other is to use the nifty systemctl service script to daemonize the process.
Prior to today, I had the same problem and ran the splunk process as root. This was foolish. If you happen to have once started splunk as root, and then successfully started splunk as the "splunk" user, you will find that your splunk login page is empty. You get the background picture, but no input controls.
Damn. Google that, nada. Damn again.
Today, I learned alot more about selinux and permissions and labels, so I investigated the "web_service" log (/opt/splunk/var/log/splunk/web_service.log) and found:
IOError: [Errno 13] Permission denied: '/opt/splunk/var/run/splunk/session-d07528932b4314e72a9f5bccd0f85fb27e8f30bd.lock'
Double damn. So I run an "ls -lZ" on that var/run/splunk directory to see what is going on, and I find the following.
drwx------. splunk splunk unconfined_u:object_r:usr_t:s0 scheduler
-rw-------. root root unconfined_u:object_r:usr_t:s0 session-d07528932b4314e72a9f5bccd0f85fb27e8f30bd
-rw-------. root root unconfined_u:object_r:usr_t:s0 session-d07528932b4314e72a9f5bccd0f85fb27e8f30bd.lock
-rw-r-----. splunk splunk system_u:object_r:usr_t:s0 splunkd.pid
There it is, my foolishness. The lock files are owned by root and not splunk. Do a quick "chown splunk.splunk" on everything in the var/run/splunk directory, and reload your splunk login page.
You should have the login now.
I have found the splunk systemctl service to be very very (very) temperamental. Sometimes it works, sometimes not. I saw someone on my google quest suggesting a "su -c blah blah" on the service commands, but that's the wrong answer. Just keep trying to get it to work, eventually something magically gives-in and cooperates. I still don't know what that something "is."
Don't run splunk as root. Don't run any web thing as root. Typically don't run anything as root. You can try changing the service config files in /etc/systemd/system, but remember to run "systemctl daemon-reload" otherwise, you will get the whiney message about changes not being reloaded.