Creating a Distributed Rackspace Load balancer Website

So, today I was taking a look at Rackspace’s Load Balancers. I wanted to put together a small tutorial how to spin up multiple cloud servers and add them to a normal HTTP load balancer. This is traditionally a use case for sites that good a lot of traffic, and/or require great redundancy both at the load balancer, and at the server level. i.e. multiple ip addresses and hardware that is failover redundant. If a server or load balancer goes down, then there is provisions to allow a new load balancer or cloud server to take over.

It’s a simple setup.

Step 1. Creating 2 or more Cloud Servers from the Control Panel.

Screen Shot 2015-10-05 at 11.38.09 AM

Create two servers, using the above process.

Step 2. I am using SSH Keys, so I provide my public ssh for SSH, (for a guide on making SSH keys, see my tutorial on this site)

Create two servers with SSH KEY AUTHENTICATION (optional), using the above process.

Screen Shot 2015-10-05 at 11.39.44 AM

Step 3. Install httpd and netcat service, in my case I am using CENTOS 7, which is a nice secure version of RHEL.
RedHat/CentOS Distributions:

yum install httpd nc

Debian/Ubuntu Distributions:

apt-get install httpd netcat

Step 4. Creating A Rackspace Load Balancer

Screen Shot 2015-10-05 at 11.43.41 AM

Step 5. Add 2 or more server nodes to the Load Balancer

Screen Shot 2015-10-05 at 11.44.09 AM

Be sure to tick the number of servers you want to add. Please note, it’s possible to add servers to the Load Balancer that aren’t part of the rackspace Network. To do that you can use the ‘add external node’ button. Please note though, that the requests between load balancer and the destination machine goes over the public network interface. Whereas requests from the load balancer to other rackspace servers will always go thru the service net by default. (that is the local 10.x.x.x IP addresses and networks).

Step 6. Configure Cloud Server Firewall Settings to Accept port 80 and (optionally) port 443

# Allow negotiating of connections on port 80 incoming (HTTP), and port 443 incoming (HTTPS)
sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT
# Allow negotiated connections replies to reach us
sudo iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

Step 7. Restart apache2 / httpd service, and check localhost on port 80

service httpd restart
curl localhost

Step 8. Configure the Load Balancer Algorithm (optional). I wanted to use a round-robin approach, each request increments the serverid. So with 3 servers. It takes 3 incremental requests for serverid 1 to serve http request. With 3 servers, i t takes 4 incremental requests for serverid 2 to serve a request twice!

Screen Shot 2015-10-05 at 11.49.41 AM

There are various different settings to use.

Step 9. All Done! You’ve configured the load balancer, and attached 2 cloud servers to the load balancer. Requests can come into the load balancer on port 80, and it sends requests to either server1 or server2 on the same port, 80. It is however possible to send requests to the cloud servers on a different port, which is covered in another article on this website. You could for instance create a server using 10 SSL certificates on a single IP, using different ports. Which could work out a lot cheaper than leasing 10 IPV4 or wrestling with your service provider for additional IP’s for SSL usage

Step 10. Test requests to the load balancer

In this setup I have 2 cloud server IP’s: and They both listen on port 80.

In this setup I have 1 load balancer IP address:

I can now connect to the load balancer running on and it forwards the connections to or If I wanted I could have added another 100 cloud servers to the load balancer, and each time I load a HTTP request to the load balancer, a different cloud server will respond. This allows very large numbers of transactions to go to a website, and for multiple servers to respond individually to seperate customers requesting thru the load balancer. In a production environment after confirming these changes I would then.

Step 11: SET DNS to point to the load balancer IP instead of cloud server IP.


Step 12: Confirm Load Balancer is working

For my purposes I am using the default www website for CENTOS 7 HTTPD /var/www/html/index.html . I changed server1 index.html to say only ‘server1’ and I changed server2 index.html to say only ‘server2’. This way I can check the load balancer is giving a different server every second request. It was:

Screen Shot 2015-10-05 at 12.01.12 PM

Screen Shot 2015-10-05 at 12.01.21 PM

Step 13: Testing for Client IP information with Netcat

tailf /var/log/httpd/access.log - - [05/Oct/2015:10:14:37 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0" - - [05/Oct/2015:10:14:39 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0" - - [05/Oct/2015:10:14:40 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0" - - [05/Oct/2015:10:14:42 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"

Presently we can see that requests reaching our servers from cloud load balancer are seeing only the load balancer IP. We need each apache httpd server to know the client IP of each request.

service httpd stop

So, I run netcat on port 80 to see the requests from the load balancer.

[root@server1 html]# nc -l 80
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Cache-Control: max-age=0
X-Forwarded-Proto: http
Accept-Language: en-US,en;q=0.5
If-Modified-Since: Mon, 05 Oct 2015 10:11:57 GMT
Via: 1.1 0A02CC2D
X-Forwarded-Port: 80
If-None-Match: "8-52158be551fbf"
Accept-Encoding: gzip, deflate

As we can see there is an X-Forwarded-For header from the load balancer which does reach the cloud server, but by default apache doesn’t know about it and doesn’t put the X-Forwarded-For variable in the Logs, only the src_ip, which is presently the load balancers IP and not our client. So we need to make a small modification to the apache2 default httpd configuration:

Step 14: Modify Apache HTTPD Log ‘combined’ configuration

cat /etc/httpd/conf/httpd.conf

Inside the above file we see the directives:


    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    CustomLog "logs/access_log" combined

The part we want to change is very small

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

We want to add the section above %{X-Forwarded-For}i. This will include the real Client_IP before the Load Balancer IP in the server logs, so it looks like so: - - [05/Oct/2015:11:01:11 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"