Building a Scalable Highly Available Web Cluster Part 1: The Load Balancing Tier

In an earlier post we discussed overall cluster configuration and now we’re going to begin actual cluster configuration with the load-balancing tier. Each load balancer will run following software:

  • HAProxy – HAProxy will facilitate the actual load balancing of traffic to web nodes behind the load balancers.
  • Keepalived – Keepalived will allow for IP failover in the event one of the load balancers fails.


NOTE: Before we begin it’s important to understand that each service must be installed and configured on both load balancers.

Step 1: Install HAProxy and Keepalived

We’ll begin by installing HAProxy and Keepalived with the following command:

[bash]yum install haproxy keepalived[/bash]

Step 2: Configure HAProxy and Keepalived

Once the software has been successfully installed it needs to be configured to function within the larger cluster environment.

Configure HAPRoxy

To configure HAProxy on the primary load balancer open /etc/haproxy/haproxy.cfg with the following command:

[bash]vim /etc/haproxy/haproxy.cfg[/bash]

After opening the file you should edit it to look similar to the example below. Note that values which differ from the default are enclosed in brackets (<>) and comment blocks explaining each change are provided inline.

[bash]
#———————————————————————
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#———————————————————————</p>
<p>#———————————————————————
# Global settings
#———————————————————————
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local0</p>
<p> chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon</p>
<p># turn on stats unix socket
stats socket /var/lib/haproxy/stats</p>
<p>#———————————————————————
# common defaults that all the ‘listen’ and ‘backend’ sections will
# use if not designated in their block
#———————————————————————
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m</p>
<p>#——————————————————————————–
# Timeout connect:
#
# Timeout connect sets the maximum time HAProxy will wait for a backend server
# to reply before sending traffic to another backend. Typically, this value is
# set slightly above a multiple of three seconds to accomodate TCP packet loss.
#
# Example value: 4s
#——————————————————————————–</p>
<p> timeout connect &lt;number of seconds HAProxy should try to connect to a backend&gt;s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000</p>
<p>#———————————————————————
# Insecure fronted to serve HTTP traffic
#———————————————————————</p>
<p>#——————————————————————————–
# Frontend:
#
# The frontend keyword can be set to any arbitrary value.
#
# Example value: http
#——————————————————————————–</p>
<p>frontend &lt;name of frontned&gt; 0.0.0.0:80</p>
<p>#——————————————————————————–
# default_backend:
#
# The default_backend keyword can be set to any arbitrary value.
#
# Example value: http
#——————————————————————————–</p>
<p>default_backend &lt;name of backend&gt;</p>
<p>#———————————————————————
# Insecure backend to load balance HTTP requests
#———————————————————————</p>
<p>#——————————————————————————–
# backend:
#
# In the case of this configuration, the values of the default_backend and backend
# keywords should match so HAProxy will be able to froward traffic to the
# correct backend.
#
# Example value:
#
# default_backend http
# backend http
#——————————————————————————–</p>
<p>backend &lt;name must match that of default_backend keyword defined above&gt;</p>
<p>balance roundrobin</p>
<p>#——————————————————————————–
# Server:
#
# The server keyword can be any arbitrary value. In the case of this particular
# configuration cookie persistence will be used to enable &quot;session stickyness&quot;
# and the cookie will be named after the server from which it originated.
# Therefore is repeated after the cookie keyword.
#
# Example value: web-srv-01
#——————————————————————————–</p>
<p>#——————————————————————————–
# Backend server IP address:
#
# The internal IP address of the backend server.
#
# Example value: 10.13.37.12
#——————————————————————————–</p>
<p>server &lt;server name&gt; &lt;backend server IP address&gt;:80 cookie check port 80
server &lt;server name&gt; &lt;backend server IP address&gt;:80 cookie check port 80
cookie JSESSIONID prefix</p>
<p>#———————————————————————
# Internal statistics backend
#———————————————————————</p>
<p>listen statistics 0.0.0.0:5674
stats enable
stats realm HAproxy Statistics</p>
<p>#——————————————————————————–
# Desired stats username/password:
#
# The authentication details that will be used to login to the HAProxy statistics
# interface.
#
# Example value: statsuser:supersecretpassword
#——————————————————————————–</p>
<p>stats auth &lt;username&gt;:&lt;password&gt;
stats uri /admin?stats
stats refresh 5s
[/bash]

Figure 1: HAProxy configuration file

HAProxy should now be configured on the second load balancer exactly as it has on the primary load balancer to ensure predictable operation in he event of a failure.

Configure Keepalived

Keepalived will provide IP failover between the load balancers by binding a floating Virtual IP Address (VIP) to the master load balancer and shifting that address to the slave should the master fail. To do so, Keepalived must be able to bind an address the system doesn’t actually own to a network adapter. This functionality is enabled via a kernel setting which is set in the /etc/sysctl.conf file. Open the /etc/sysctl.conf file with the following command

[bash]vim /etc/sysctl.conf[/bash]

Once you’ve opened the file, simply add net.ipv4.ip_nonlocal_bind = 1 to the end of the file then reboot the system to ensure the new setting is persistent.

Now that the OS has been configured to allow non-local bind we can move on to configuring Keepalived proper. We’ll begin by opening /etc/keepalived/keepalived.conf on LB-01 with the following command:

[bash]vim /etc/keepalived/keepalived.conf[/bash]

The file should be edited to look similar to the file below. As with the configuration file for HAProxy, important values are enclosed within braces (<>) and inline explanations are provided.

[bash]</p>
<p>vrrp_script chk_haproxy {
script &quot;killall -0 haproxy&quot;</p>
<p>#——————————————————————————–
# Interval:
#
# The amount of time each load balancer waits between health checks.
#
# Example value: 2
#——————————————————————————–</p>
<p>interval &lt;number of seconds between keepalived health checks&gt;
weight 2
}</p>
<p>#——————————————————————————–
# Name of VRRP instance:
#
# The desired name for this VRRP instnace.
#
# Example value: load_balancer
#——————————————————————————–</p>
<p>vrrp_instance &lt;instance name&gt; {</p>
<p>interface eth0
state MASTER
virtual_router_id 01</p>
<p>#——————————————————————————–
# Priority:
#
# The priority value should be set to 101 on the master and 100 on the slave.
#——————————————————————————–</p>
<p>priority 101</p>
<p>#——————————————————————————–
# virtual_ipaddress:
#
# The virtual_ipaddress is the IP address that will float between the load
# balancers (depending on health state) and will be the address on which any
# services that exist behind the load balancers will be accessed.
#
# Example value: 10.13.37.10
#——————————————————————————–</p>
<p>virtual_ipaddress {
&lt;virtual IP address&gt;
}</p>
<p>track_script {
chk_haproxy
}
}
[/bash]

Figure 2: Keepalived configuration file (LB-01)

Keepalived should be configured on the secondary load balancer exactly as it has been on the primary load balancer with one exception, the priority value. As shown in the below configuration file, the priority value should be set to 100 on the backup (secondary) load balancer.

[bash]</p>
<p>vrrp_script chk_haproxy {
script &quot;killall -0 haproxy&quot;</p>
<p>#——————————————————————————–
# Interval:
#
# The amount of time each load balancer waits between health checks.
#
# Example value: 2
#——————————————————————————–</p>
<p>interval &lt;number of seconds between keepalived health checks&gt;
weight 2
}</p>
<p>#——————————————————————————–
# Name of VRRP instance:
#
# The desired name for this VRRP instnace.
#
# Example value: load_balancer
#——————————————————————————–</p>
<p>vrrp_instance &lt;instance name&gt; {</p>
<p>interface eth0
state MASTER
virtual_router_id 01</p>
<p>#——————————————————————————–
# Priority:
#
# The priority value should be set to 101 on the master and 100 on the slave.
#——————————————————————————–</p>
<p>priority 100</p>
<p>#——————————————————————————–
# virtual_ipaddress:
#
# The virtual_ipaddress is the IP address that will float between the load
# balancers (depending on health state) and will be the address on which any
# services that exist behind the load balancers will be accessed.
#
# Example value: 10.13.37.10
#——————————————————————————–</p>
<p>virtual_ipaddress {
&lt;virtual IP address&gt;
}</p>
<p>track_script {
chk_haproxy
}
}
[/bash]

Figure 3: Keepalived configuration file (LB-02)

Step 3: Configure Iptables

The Keepalived service on the load balancers uses VRRP (multicast) to determine node health so we need to create the appropriate firewall rules. We’ll also create the firewall rule required to allow connections on port 80 so the load balancers can accept and route traffic to backend web servers.

Let’s first look at the out-of-box iptables rule set:

[bash]</p>
<p>*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -j REJECT –reject-with icmp-host-prohibited
COMMIT</p>
<p>[/bash]

Figure 4: Out-of-box iptables configuration

There are a couple different ways to manipulate firewall rules under CentOS, but we’re going to directly edit the rules file used by iptables since running the iptables command with the -A flag will append the rule to the end of the chain (after any drop rules that may be present in the chain) which would prevent the rule set from correctly delating with traffic. We’ll begin by opening the iptables rule file with the following command:

[bash]vim /etc/sysconfig/iptables[/bash]

Now we’ll add the rules we need to allow VRRP and HTTP:

[bash]</p>
<p>*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp –dport 80 -j ACCEPT # Newly added rule to allow traffic on port 80
-A INPUT -s -d 224.0.0.8/8 -p vrrp -j ACCEPT # Newly added rule to allow VRRP between the load balancers.
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -j REJECT –reject-with icmp-host-prohibited
COMMIT</p>
<p>[/bash]

Figure 5: Modified iptables rule set (LB-01)

Iptables should be configured identically on the backup load balancer with one exception, the source address in the VRRP rule should be the IP address of the primary load balancer as shown in the example below:

[bash]
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp –dport 80 -j ACCEPT # Newly added rule to allow traffic on port 80
-A INPUT -s -d 224.0.0.8/8 -p vrrp -j ACCEPT # Newly added rule to allow VRRP between the load balancers.
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -j REJECT –reject-with icmp-host-prohibited
COMMIT
[/bash]

Figure 6: Modified iptables rule set (LB-01)

Once you have edited the firewall configuration on each node you should save and close the file with :wq and apply the new firewall rules on each node with the following commands:

[bash]</p>
<p>service iptables restart
service iptables save</p>
<p>[/bash]

Step 4: Start Services and Set Service Persistence

Now that we’ve configured HAproxy and Keepalived we’ll start the services and configure them to start at boot with the following commands:

[bash]</p>
<p>service haproxy start
chkconfig haproxy on</p>
<p>service keepalived start
chkconfig keepalived on</p>
<p>[/bash]

Step 5: Test IP Failover

Now that we’ve configured HAProxy, Keepalived and iptables we can test our configuration to ensure Keepalived will detect a problem on either load balancer and correctly bind to the VIP in the event of a failure. We’ll begin by running the following command and LB-01 (the active load balancer):

[bash]ip addr sh eth0[/bash]

The output of the above command should look similar to the following example:

[bash]</p>
<p>[root@LB-01 ~]# ip addr sh eth0
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:26:b1:88 brd ff:ff:ff:ff:ff:ff
inet 10.13.37.2/24 brd 10.13.37.255 scope global eth0
inet 10.13.37.10/32 scope global eth0
inet6 fe80::a00:27ff:fe26:b188/64 scope link
valid_lft forever preferred_lft forever</p>
<p>[/bash]

Figure 7: ip addr sh eth0 output from LB-01 (load balancing tier healthy)

As you can see from the output above, LB-01 is the active load balancer and is bound to both 10.13.37.2 (the local IP of the load balancer) and 10.13.37.10 (the floating VIP defined in the Keepalived configuration). We’ll now take a look at the output of the same command of LB-02 (the passive load balancer)

[bash]</p>
<p>[root@LB-02 ~]# ip addr sh eth0
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:c9:cd:8d brd ff:ff:ff:ff:ff:ff
inet 10.13.37.4/24 brd 10.13.37.255 scope global eth0
inet6 fe80::a00:27ff:fec9:cd8d/64 scope link
valid_lft forever preferred_lft forever</p>
<p>[/bash]

Figure 8: ip addr sh eth0 output from LB-02 (load balancing tier healthy)

As you can see, LB-02 is only bound to 10.13.37.4 (the local IP address of the load balancer) since LB-01 is functioning normally. What happens when something goes wrong with LB-01 and it falls over and bursts into flames? Let’s test what happens by shutting down LB-01 with the following command:

[bash]shutdown -P now[/bash]

With LB-01 offline we can now run ip addr sh eth0 again on LB-02 and look for any differences in the output:

[bash]</p>
<p>[root@LB-02 ~]# ip addr sh eth0
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:c9:cd:8d brd ff:ff:ff:ff:ff:ff
inet 10.13.37.4/24 brd 10.13.37.255 scope global eth0
inet 10.13.37.10/32 scope global eth0
inet6 fe80::a00:27ff:fec9:cd8d/64 scope link
valid_lft forever preferred_lft forever</p>
<p>[/bash]

Figure 9: ip addr sh eth0 output (LB-01 failed)

As you can see, Keepalived detected LB-01 failed and bound to the floating VIP to eth0. Cool!

Conclusion

If you have followed the article, and haven’t encountered any errors along the way, you should now have a pair of functional load balancers. Load balancers are great in and of themselves, but we have nothing to actually load balance at the moment. In part two of this series we’ll bring up the web server/cacheing tier. As always if you have any questions or concerns please don’t hesitate to leave a comment.

//J

Leave a Reply

Your email address will not be published. Required fields are marked *