Before you say anything I know that Splunk 6.0.3 has been replaced, and that the upgrade is easy. Pooling is deprecated, but this article is really about a POC and LDAP integration more than the upgrade.
The title should really be OpenLDAP with Splunk 6.0.3, once I’ve done this I will do a blog post about Splunk 6.4.3 and OpenLDAP because IHAC for whom we just went through an exercise of setting up MS LDAP integration with Splunk 6.4.3.
The idea is to have an ubuntu 14.04 search head pool running Splunk 6.0.3 and FreeBSD 10 indexer cluster running Splunk 6.0.2, once this is done, we are going to upgrade both to Splunk 6.3.2, which will convert the search head pool to a search head cluster.
Search head pooling is what clustering was called prior to 6.2, as we are running search heads with 6.0.3 we are going to set up search head pooling first.
The three servers are called:
Ubuntu-14.04-splunk-SH1
Ubuntu-14.04-splunk-SH2
Ubuntu-14.04-splunk-SH3
Set the appropriate names in the following two files:
sudo vi /etc/hostname
sudo vi /etc/hosts
Configure static IPs (refer to diagram for info).
sudo vi /etc/network/interfaces
iface eth0 inet static
address 10.0.0.24/5/6
netmask 255.255.255.0
address 10.0.0.24/5/6
netmask 255.255.255.0
gateway 10.0.0.1
dns-nameservers 75.75.76.76
dns-nameservers 75.75.75.75
dns-nameservers 75.75.75.75
dns-nameservers 8.8.8.8
1. Set up a shared storage location accessible to each search head.
NFS for *NIX, for this exercise the NFS mount point is going to be on SH1 and it is going to be accessible by SH2 and SH3.
The mount point is going to be called /S2HPS (Shared Search Head Pool Storage)
apt-get install nfs-kernel-server
mkdir -p /export/S2HPS
mount --bind /home/michaelpeters/splunk /export/S2HPS
- To save us from retyping this after every reboot we add the followingline to /etc/fstab
/home/michaelpeters/splunk /export/S2HPS none bind 0 0
Edit exports:
sudo vi /etc/exports
/export 10.10.0.0/24(rw,fsid=0,insecure,no_subtree_check,async)
/export/S2HPS 10.0.0.0/24(rw,nohide,insecure,no_subtree_check,async)
sudo exportfs -ra
sudo service nfs-kernel-server restart
Then NFS client installation on SH2 and SH3
Install the required packages:
-
# apt-get install nfs-common
On the client we can mount the complete export tree with one command:
-
# mount -t nfs -o proto=tcp,port=2049 10.0.0.24:/export/S2HPS /mnt/S2HPS
Configure SH2 and SH3 to auto mount the NFS mount on boot
add the following entry into /etc/fstab as root
10.0.0.24:/export/S2HPS /mnt/S2HPS nfs defaults 0 0
2. Configure each individual search head.
Set up the cluster
Replication factor 3, Search Factor 2, Security Key=changeme
Enable Master Node 10.0.0.124
Enable Peer Nodes 10.0.0.125, and 10.0.0.126
Master IP address: https://10.0.0.124
Master Port 8089
Peer replication port 8080
Security Key=changeme
on the two Peers:
$ pwd
/usr/home/michaelpeters/splunk/etc
$ mv instance.cfg instance.cfg.bak
3. Stop the search heads.
4. Enable pooling on each search head.
5. Copy user and app directories to the shared storage location.
6. Restart the search heads.
Test the search head set up
-
splunk pooling validate -
splunk pooling display
Configure splunk to start automatically on each box
sudo ./splunk/bin/splunk enable boot-start -user michaelpeters
LDAP server set up
LDAP IP is 10.0.0.33
LDAP admin is admin, password = changeme
When adding the first user if the below is not fixed the password cannot be added via phpldapadmin
change
password_hash in /usr/share/phpldapadmin/lib/TemplateRender.phpon line 2469 to password_hash_custom.I played around with the SPLUNK LDAP configuration but what worked was :
Which really used the defaults for everything, but I also incorporated the info from:
The configuration does not use SSL, so the communication between the LDAP server and the SPLUNK instance is not secure and vulnerable to sniffing.
Once the pool is set up a load balancer will be set up at the front end, to this end either Apache in reverse proxy mode or squid can be used, we are going to use squid:
Initially we have only two VMs running ubuntu server 14.04 and FreeBSD 10.
The Ubuntu VMs are going to be running locally on my laptop
The FreeBSD VMs are going to be running remotely on my mac mini
Next steps:
There is going to be a forwarder on a Windows 10 box
Instead there is going to be the Ubuntu 14.04 LDAP server configured as a forwarder
There is going to be a data generator
The reason for the above thee data sources is that we want to test both the Front end loadbalancer set up for the search head pooling but also the forwarder clustering set up to the indexing cluster
There is going to be a deployment server (SPLUNK) which is going to be migrated to have a deployer installed on it this can be CentOS 5.11
The deployment server is going to be left until the above is completed
Links
6.3 upgrade:
Search Head Pooling blog
Additional exercises:
set up LDAP with NFS
Once done we are going to do all the same using the latest version of CentOS as this is the same as RedHat Enterprise Linux.
