Using SNI with Rackspace Cloud Load Balancer and adding upto 20 SSL Certificates on single LB

Posted on by

This is going to be a short and dirty documentation on how to add multiple SSL certificates to a Rackspace Load Balancer.

1. Authorise with rackspace auth api (Get a token with user and api key credentials)
x-auth-key is apikey and x-auth-user is the mycloud username

curl -D - -H "x-auth-user: myusername" -H "x-auth-key: 1c989d8f89dfd87f3df3dff3d6f7fgf" https://auth.api.rackspacecloud.com/v1.0


HTTP/1.1 204 No Content
Server: nginx
Date: Thu, 19 Nov 2015 15:41:38 GMT
Connection: keep-alive
X-Storage-Token: AAA98345kdfg893DFGDF43iudng39dfgjkdfgDFI$JUIDFJGDFJGDFGDJJHDFGJHIfdg34dfgkdfjgiodfgiodfDFGDdg323
X-Storage-Url: https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_1001001
X-NewRelic-App-Data: PxQGUF9aDwETVlhSBQUP
X-CDN-Management-Url: https://cdn3.clouddrive.com/v1/MossoCloudFS_1001001
X-Auth-Token: AAA98345kdfg893DFGDF43iudng39dfgjkdfgDFI$JUIDFJGDFJGDFGDJJHDFGJHIfdg34dfgkdfjgiodfgiodfDFGDdg323
vary: Accept, Accept-Encoding, X-Auth-Token, X-Auth-Key, X-Storage-User, X-Storage-Pass, X-Auth-User
Cache-Control: s-maxage=86319
Front-End-Https: on

Now you can copy and paste the X-Auth-Token. It is needed for the next step

2. Configure the JSON file to upload an additional certificate and private key via API for a domain hostname. Here I am configuring domain.com

file: lb.json

{
  "certificateMapping": {
     "hostName": "domain.com",
     "certificate": "-----BEGIN CERTIFICATE-----\nMIIC/TCCAeWgAwIBAgIJAOjRMYJKDeryMA0GCSqGSIb3DQEBBQUAMBUxEzARBgNV\nBAMMCmRvbWFpbi5jb20wHhcNMTUxMTE5MTQzMjE3WhcNMjUxMTE2MTQzMjE3WjAV\nMRMwEQYDVQQDDApkb21haW4uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEAvHTjzWQchX+Gyl/No+ABR9R+F65rJmEPBEutjgWUynOir7ZYu5vmFol8\nhF054W5Xv3Ii4oYJjDJingOqQUBBxJD4jXx8H79y04JGXl8BBrG7azbRbowc4HoP\nRUiVTNaCPgYAGTreiRXmYKb/beotlGDvl0HQQLeDh4iq1X1E8R/lkFRHVAu0rEgC\nIeuJZ2L3Qu06A5yTCwdTJnZmviLmuDQtkfLDqTA8N67U8zjBgKGsj9t7GDSQ7zGp\n6JbTSJXqsXvd7XMLm2Ns2UelVUToxBTwgOIBn0XzZLCIOIlbIn0LHBk8oYEA4JDF\n1mXeqdsFOCtYvFcQBoUihiDjwDdTNQIDAQABo1AwTjAdBgNVHQ4EFgQU1wBZxNte\n9Q//UOl7ZMUvtsXghPEwHwYDVR0jBBgwFoAU1wBZxNte9Q//UOl7ZMUvtsXghPEw\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAUNM56u/cc56ESZY4gubX\nh0UQ8TjVbV2G4EkbBkNnm7RgNK48lFIxc55tshawhdN01JH5ZIgB1RvO1/lqouVs\nJrXwnPULBb4M5FcrjjBVu3bIvOjAUVDogOm7pKP/hJALM9CWMuZcXr5C+sYFczaB\nA7uDuMuQoTZBIGF1NyzfO7vmHT5QbEA/1ZYISWrVFNt8g2oxJY+jdgKacxVujWIs\nFpuiCCdvFVI05wCjj3C8BIN/EAcRIqe5gwr5oI+AtwK7fjK5K47/sREMI+W6Bj1w\nZEDz92S+dNtoSPJTBWiIQFLslTPiaDAu1EjJO1+YRXG7LANdxpQrogvDG1l9VpDW\nRg==\n-----END CERTIFICATE-----",
  "privateKey": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEAvHTjzWQchX+Gyl/No+ABR9R+F65rJmEPBEutjgWUynOir7ZY\nu5vmFol8hF054W5Xv3Ii4oYJjDJingOqQUBBxJD4jXx8H79y04JGXl8BBrG7azbR\nbowc4HoPRUiVTNaCPgYAGTreiRXmYKb/beotlGDvl0HQQLeDh4iq1X1E8R/lkFRH\nVAu0rEgCIeuJZ2L3Qu06A5yTCwdTJnZmviLmuDQtkfLDqTA8N67U8zjBgKGsj9t7\nGDSQ7zGp6JbTSJXqsXvd7XMLm2Ns2UelVUToxBTwgOIBn0XzZLCIOIlbIn0LHBk8\noYEA4JDF1mXeqdsFOCtYvFcQBoUihiDjwDdTNQIDAQABAoIBAQCSEJr7d0tv4P6s\n3gI5sIXtkXHFkwczcOi9sJYszICdRXDjdZZimpuD/j3HLaaN5gMWvDTzk2XVBrxO\nspKEDnSrEJ3Es6ZUyQMLkh5OSJ43/QtBNvSuFOTQy2oIjhBBxMSfo/DxnSIb6CBt\n6yFwpJ99MICioHzznAjSxId7/qKvq294emBGwpyP6JbCEtrM6rsnBO4J/uHUDLRj\nlU0zLFwFHNQnhnfIuxOoUZthyCSzZgUquC7C52qIPTZxqCydSi045pDoymn6pT43\n5YdafzWarmEqBGcyqDOyjOz01IEicrmFW7e2+DICIOTOvTSeFQtHbO4Rn2VE2V+x\nGNJY3DoFAoGBAORqB6gFlLUKBXdmP1VcEifjwcVtBaY9QwehbH8En6O0N1t5bKFx\nTBaShm2El+7UCeeSz9hx3vmV/4gn9amJnu6stOEUfjbfxe6mw8OtR13g5iSAI9TQ\nXesf1HoCrUsljzAPvBAKxWSQl9e6fYBxmB1IvFvd4n9uvoNWr/lOfbe3AoGBANM3\neddZYHBB0PhgiJ9aq7QkgqUSdv5JlBdtGdPDr3cpIx9QmXMtf+wc8vZ6CSvC3EIn\npADRt3QAIzxQLpXb3ADjBCwwsFCu27IXlVkvxD+yvqaLbAjB/LgbKqt5wR6YAarj\nDQzNzxhGvrCS+CvYSKospY6UK5+V0nuhuPVcuJRzAoGAAPHLTE+RmNoMwbyjgGfc\nD1wqvfVAc7qHH230c+YB/vxMyk0LPPOp++HpOmS0+CDaVaHOyDdYU7HiF58KrgPK\nq3P9X3zlNLbiK6V248VAqUu3x+jbvRKLgOBl0YdXThs+p1U5Utuoi0zpw9Oal0Bg\n/6YAWWTmfd5oXUSrf51qeasCgYEAgMahBZgbgTXPh6+rfKTWbQWZlbU1UYJgxQui\npIb5cwhkvpHwjNWf2cAorffnoYOzsK3kgw9Z72KqGPq1/G5Iq0293Idu6DJEBkf0\nqaTC3SdIr9fvbUOApmsBz/xyrwl0ctDtwvG0IxP27UceAfVjEEYaRly2YB0DcJdA\nYnA+pVsCgYEAoHfkw/ZPmB7r8LesF0+N93AErJ/IiPoCBFNKijVDplzLQbMeWyxL\njcnFdq8vQT0Os4qzRNCR5QbMcprJIh4LC96OIlGWz5NhKCWbGsKxA8N7YoWGYy9Z\nmRkVP6peBU2cGdXRWjCrxkKR+uJM9BCG0Ix3BOPy29nWaCEl+5wjBEc=\n-----END RSA PRIVATE KEY-----"
  }

3. Call API to add certificatemapping json lb hostname configuration file. This just allows example.com to have SSL on the Load Balancer.
(you can add up to 20 Domains). It’s lots cheaper and not as hard as I might have initially thought!!

curl -v -H "X-Auth-Token: $TOKEN" -d @lb.json -X POST -H "content-type: application/json"  https://lon.loadbalancers.api.rackspacecloud.com/v1.0/1001001/loadbalancers/157089/ssltermination/certificatemappings

It’s also possible to update the Load Balancer Certificates via the API, please see https://developer.rackspace.com/docs/cloud-load-balancers/v1/developer-guide/#update-certificate-mapping for more information

4. Confirm the certificate mappings are added (please note 1001011 is the customer DDI and 157090 is the Load Balancer ID).

curl -v -H "X-Auth-Token: $TOKEN" -X GET https://lon.loadbalancers.api.rackspacecloud.com/v1.0/1001011/loadbalancers/157090/ssltermination/certificatemappings


< HTTP/1.1 200 OK
< Content-Type: application/json
< Via: 1.1 Rackspace Cloud Load Balancer API v1.25.11 (Repose/2.11.0)
< Content-Length: 83
< Date: Thu, 19 Nov 2015 15:49:24 GMT
* Server Jetty(8.0.y.z-SNAPSHOT) is not blacklisted
< Server: Jetty(8.0.y.z-SNAPSHOT)
<
* Connection #0 to host lon.loadbalancers.api.rackspacecloud.com left intact
{"certificateMappings":[{"certificateMapping":{"id":999,"hostName":"domain.com"}}]}

You may note that the lb.json file has the certificate all on one line! how to do this? It's not that hard. Here is how I did it:

cat domain.com.cert | sed ':a;N;$!ba;s/\n/\\n/g'
cat domain.com.key  | sed ':a;N;$!ba;s/\n/\\n/g'
-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEAvHTjzWQchX+Gyl/No+ABR9R+F65rJmEPBEutjgWUynOir7ZY\nu5vmFol8hF054W5Xv3Ii4oYJjDJingOqQUBBxJD4jXx8H79y04JGXl8BBrG7azbR\nbowc4HoPRUiVTNaCPgYAGTreiRXmYKb/beotlGDvl0HQQLeDh4iq1X1E8R/lkFRH\nVAu0rEgCIeuJZ2L3Qu06A5yTCwdTJnZmviLmuDQtkfLDqTA8N67U8zjBgKGsj9t7\nGDSQ7zGp6JbTSJXqsXvd7XMLm2Ns2UelVUToxBTwgOIBn0XzZLCIOIlbIn0LHBk8\noYEA4JDF1mXeqdsFOCtYvFcQBoUihiDjwDdTNQIDAQABAoIBAQCSEJr7d0tv4P6s\n3gI5sIXtkXHFkwczcOi9sJYszICdRXDjdZZimpuD/j3HLaaN5gMWvDTzk2XVBrxO\nspKEDnSrEJ3Es6ZUyQMLkh5OSJ43/QtBNvSuFOTQy2oIjhBBxMSfo/DxnSIb6CBt\n6yFwpJ99MICioHzznAjSxId7/qKvq294emBGwpyP6JbCEtrM6rsnBO4J/uHUDLRj\nlU0zLFwFHNQnhnfIuxOoUZthyCSzZgUquC7C52qIPTZxqCydSi045pDoymn6pT43\n5YdafzWarmEqBGcyqDOyjOz01IEicrmFW7e2+DICIOTOvTSeFQtHbO4Rn2VE2V+x\nGNJY3DoFAoGBAORqB6gFlLUKBXdmP1VcEifjwcVtBaY9QwehbH8En6O0N1t5bKFx\nTBaShm2El+7UCeeSz9hx3vmV/4gn9amJnu6stOEUfjbfxe6mw8OtR13g5iSAI9TQ\nXesf1HoCrUsljzAPvBAKxWSQl9e6fYBxmB1IvFvd4n9uvoNWr/lOfbe3AoGBANM3\neddZYHBB0PhgiJ9aq7QkgqUSdv5JlBdtGdPDr3cpIx9QmXMtf+wc8vZ6CSvC3EIn\npADRt3QAIzxQLpXb3ADjBCwwsFCu27IXlVkvxD+yvqaLbAjB/LgbKqt5wR6YAarj\nDQzNzxhGvrCS+CvYSKospY6UK5+V0nuhuPVcuJRzAoGAAPHLTE+RmNoMwbyjgGfc\nD1wqvfVAc7qHH230c+YB/vxMyk0LPPOp++HpOmS0+CDaVaHOyDdYU7HiF58KrgPK\nq3P9X3zlNLbiK6V248VAqUu3x+jbvRKLgOBl0YdXThs+p1U5Utuoi0zpw9Oal0Bg\n/6YAWWTmfd5oXUSrf51qeasCgYEAgMahBZgbgTXPh6+rfKTWbQWZlbU1UYJgxQui\npIb5cwhkvpHwjNWf2cAorffnoYOzsK3kgw9Z72KqGPq1/G5Iq0293Idu6DJEBkf0\nqaTC3SdIr9fvbUOApmsBz/xyrwl0ctDtwvG0IxP27UceAfVjEEYaRly2YB0DcJdA\nYnA+pVsCgYEAoHfkw/ZPmB7r8LesF0+N93AErJ/IiPoCBFNKijVDplzLQbMeWyxL\njcnFdq8vQT0Os4qzRNCR5QbMcprJIh4LC96OIlGWz5NhKCWbGsKxA8N7YoWGYy9Z\nmRkVP6peBU2cGdXRWjCrxkKR+uJM9BCG0Ix3BOPy29nWaCEl+5wjBEc=\n-----END RSA PRIVATE KEY-----

Notice the extra \n's after the processing.

Don't be intimidated by the sed line, it just replaces the \n newline with the character \n instead, so the json file is easier to lay out the cert as a 'string'.

Important notes on SNI:

We know we can add certificate mappings on the Load Balancer.
the Load Balancer has been configured for Allowing secure and insecure traffic, Port 443. SSL is terminated at the load balancer. This is what is known as OFFLOADING, it just means the SSL encryption is seen at the load balancer. Behind the load balancer, there is no encryption between it and the server. This allows the SNI hostname to be forwarded to the server, without it being in an encrypted form within the TCP packet.

5. Now lets install apache2 and configure some virtualhosts, at the most basic level. This is for an example and not a perfect setup

apt-get update
apt-get install httpd

vi /etc/apache2/httpd.conf



ServerName example.com
Documentroot /var/www/example.com/html




ServerName domain.com
Documentroot /var/www/domain.com/html



mkdir -p /var/www/domain.com/html
mkdir -p /var/www/example.com/html
echo 'example.com page body testing' > /var/www/example.com/html/index.html
echo 'domain.com page body testing' > /var/www/domain.com/html/index.html

vi /etc/apache2/apache2.conf

add one line in the file like:

Include /etc/apache2/httpd.conf

I just like to configure apache2 this way.

/etc/init.d/apache2 restart

6. Confirm both websites are working thru LB with SNI

# Curl domain
$ curl domain.com
domain.com page body testing

# curl domain 2
$ curl example.com
example.com page body testing

# curl IP address
curl https://194.213.79.117
someotherdefaultpage

# what happened when curling the IP address? Well..There was no TCP servername/hostname forwarded in the header for SNI support to detect the domain x-forwarded-for

# Lets provide the header
curl https://194.213.79.117 -H "host: example.com"

Testing SSL on the hostnames

openssl s_client -connect domain.com:443
openssl s_client -connect domain.com:443 -host domain.com
openssl s_client -connect domain.com:443 -servername domain.com

List all Cloud Server Details thru the API

Posted on by

Well, this one is a bit cheeky because I borrowed it from a colleague of mine David Coon. Thanks David, I appreciate your assistance!

#!/bin/bash


auth() {
    read -p "What is your Account Number: " ddi
    read -p "Whats your username:" username    
    read -p "Whats your APIkey:" APIkey
    read -p "Which Datacenter are your servers in? " dc
}

token() {
    
    token=`curl -s https://identity.api.rackspacecloud.com/v2.0/tokens -X POST \
    -d '{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"'$username'", "apiKey":"'$APIkey'"}}}' \
    -H "Content-Type: application/json" | python -m json.tool  | sed -n '/expires/{n;p;}' |sed -e 's/^.*"id": "\(.*\)",/\1/'`
    echo "Your API Token is ---->  $token"
}

listservers() {
    curl -s -H "X-Auth-Token: $token" "https://$dc.servers.api.rackspacecloud.com/v2/$ddi/servers" | python -m json.tool
}

getservers() {
    read -p "What is the server id?" id
    curl -s -H "X-Auth-Token: $token" "https://$dc.servers.api.rackspacecloud.com/v2/$ddi/servers/$id" | python -m json.tool
}

auth
token
listservers
getservers

Installing Nova Agent Linux on Xen Guest VM

Posted on by

So, sometimes every now and then a customer wants to use a custom image with our services. The thing is for the build to succesfully complete and the VM to get networking, it needs to be able to communicate with lil ole nova-agent.

PLEASE ALSO SEE http://www.haxed.me.uk/index.php/2016/10/06/rackspace-cloud-server-not-coming-building/

1. Download the nova-agent-linux

cd ~/
mkdir nova-agent
cd nova-agent
wget http://boot.rackspace.com/files/nova-agent/nova-agent-Linux-x86_64-1.39.0.tar.gz

2. Extract and run installer script

tar xzf nova-agent-Linux-x86_64-1.39.0.tar.gz

3. Inject LSB headers into the script (if not already there)

 
sed '1i### BEGIN INIT INFO\n# Provides: Nova-Agent\n# Required-Start: $remote_fs $syslog\n# Required-Stop: $remote_fs $syslog\n# Default-Start: 2 3 4 5\n# Default-Stop: 0 1 6\n# Short-Description: Start daemon at boot time\n# Description: Enable service provided by daemon.\n### END INIT INFO\n' /usr/share/nova-agent/1.39.0/etc/generic/nova-agent > /usr/share/nova-agent/1.39.0/etc/generic/nova-agent.lsb

4. Move the init script in place and make it executable

cp -av /usr/share/nova-agent/1.39.0/etc/generic/nova-agent.lsb /etc/init.d/nova-agent
chmod +x /etc/init.d/nova-agent

5. Set the script to start automatically in the event of a reboot.

# RHEL, CentOS, Fedora, OpenSuse
chkconfig nova-agent on

# Debian, Ubuntu
update-rc.d -f nova-agent defaults

PLEASE ALSO SEE: http://www.haxed.me.uk/index.php/2016/10/06/rackspace-cloud-server-not-coming-building/

Testing if Nova Agent is available on server when an image build is not getting networking

Posted on by

So, we occasionally get customers who are having issues creating a new server from an image of a previous server. Normally this is caused by the nova-agent not being set to start on boot, or the xe-linux-distribution being missing from the VM. It’s possible to check whether a virtual machine is configured right and I put together this little piece with the help of my colleague and friend Zoltan.

1. Checking if Nova Agent is installed and can be started

# /etc/init.d/nova-agent start

2. Check if nova-agent and xe-linux-distribution is running on VM

# ps auxf | grep nova
# ps auxf | grep xe-daemon

If processes called nova-agent or xe-daemon return then you know they are running OK.

3. Ensure that both services do start during boot

# chkconfig nova-agent on
# chkconfig xe-linux-distribution on

For Debian and Ubuntu Systems you may need to use

update-rc.d -f nova-agent defaults

Once you confirm that these services are running it’s safe to take an image, and create a new VM with it. These 2 processes need to be running because when the new server is built the way that the VM gets its networking set is using xenstore and novaagent to retrieve and set the network interfaces file with correct ip, subnet and gateway.

– A

Manually Creating a Bootable CBS using NOVA

Posted on by

A customer was getting a bad error: : Block Device Mapping is Invalid.

It was because the cbs wasn’t building in time from the image , and the build was timing out. So the solution was pretty simple. Add the CBS first:


 supernova customer volume-create 55 --volume-type=SSD --display-name=starating --image-id=5674345-dfgegdf-34553531123

Oh, thanks Aaron dude. You rock.

Deleting All the Files in a Cloud Container

Posted on by

Hey. So if only I had a cake for every customer that asked if we could delete all of their cloud files in a single container for them (i’d be really really really fat so maybe that is a bad idea). A dollar though, now there’s a thought.

On that note, here is a dollar. Probably the best dollar you’ll see today. You could probably do this with php, bash or swiftly, but doing it *THIS* way is also awesome, and I learnt (although some might say learned) something. Here is how I did it. I should also importantly thank Matt Dorn for his contributions to this article. Without him this wouldn’t exist.

Step 1. Install Python, pip 

yum install python pip
apt-get install python pip

Step 2. Install Pyrax (rackspace Python Openstack Library)

pip install pyrax

Step 3. Install Libevent 

curl -L -O https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
tar xzf libevent-2.0.21-stable.tar.gz
cd libevent-2.0.21-stable
./configure --prefix="$VIRTUAL_ENV"
make && make install
cd $VIRTUAL_ENV/..

Step 4. Install Greenlet and Gevent


pip install greenlet
pip install gevent

Step 5. Check gevent library loading in Python Shell 

python
import gevent

If nothing comes back, the gevent lib works OK.

Step 6. Create the code to delete all the files

#!/usr/bin/python
# -*- coding: utf-8 -*-
from gevent import monkey
from gevent.pool import Pool
from gevent import Timeout
monkey.patch_all()
import pyrax

if __name__ == '__main__':
    pool = Pool(100)
pyrax.set_setting('identity_type', 'rackspace')
pyrax.set_setting('verify_ssl', False)
# Rackspace Credentials Go here, Region LON, username: mycloudusername apikey: myrackspaceapikey. 
pyrax.set_setting('region', 'LON')
pyrax.set_credentials('mycloudusername', 'myrackspaceapikey')

cf = pyrax.cloudfiles
# Remember to set the container correctly (which container to delete all files within?)
container = cf.get_container('testing')
objects = container.get_objects(full_listing=True)


def delete_object(obj):

# added timeout of 5 seconds just in case

    with Timeout(5, False):
        try:
            obj.delete()
        except:
            pass


for obj in objects:
    pool.spawn(delete_object, obj)
pool.join()

It’s well worth noting that this can also be used to list all of the objects as well, but that is something for later…

Step 7. Execute (not me the script!)

The timeout can be adjusted. And the script can be run several times to ensure any missed files are retried to be deleted.

Retrieving Image Meta Data and setting vm_mode

Posted on by

So, today we had a customer that was using a custom VHD/VDI with his server and it wasn’t working. I knew it would be one of the 3 things:

1) incorrect vm_mode flag
2) Image size too big for flavors ‘min disk’
3) image using journaling

As it turned out this customer was using journaling. However, if it was a PV HVM type, here is how I would correctly set the mode

supernova customer image-meta vfd09a81-g431-4279-9467-5e4284944b53 set vm_mode=hvm

Pretty simple fix.