Category Archives: Cloud

Resetting Meta data of a RAX Rackspace Xen Server

Posted on by

At work we had some customers complaining of metadata not being removed on their servers.


nova --os-username username --os-password apigoeshere meta uuidgoeshere delete rax:reboot_window

It was pretty simple to do as a one liner right.

But imagine we have a list.txt full of 100 servers that need clearing for an individual customer, that would be a nightmare to do manually like above. so we can do it like:

for server in $(cat list.txt); do nova --os-username username --os-password apikeygoeshere meta $server delete rax:reboot_window; done

Now that is pretty cool. And saved me and my colleagues a lot of time.

Testing the consistency of response time on a website

Posted on by

So, we had some customers today complaining about inconsistent page load times. So I taken a look at the hypervisor they were on and I could see it was really quite busy. In the sense that all 122GB of RAM available was being used by server instances. Ironically though it wasn’t that busy, but I live-migrated the customer anyway to a much quieter server, but the customer saw no change whatsoever.

In this case it indicated already that it wasn’t a network infrastructure or hardware issue and likely the increase in latency they saw over the last so many days was being caused by something else. Most likely the growing size of their database, not being reflected by their static amount of ram and the variables set for their tablesize cache, and etc in MySQL.

So my friend kindly put together this excellent oneliner. Check it out!

$ for i in $(seq 50); do curl -sL http://www.google.com/ -o /dev/null -w %{time_total}\\n; sleep 1; done
0.698
0.493
0.365
0.293
0.326
0.525
0.342
0.527
0.445
0.263
0.493

Pretty neat eh

Using configdrive cloud-config to execute commands post server creation

Posted on by

A lot of customers might want to setup automation, for installing common packages and making configurations for vanilla images. One way to provide that automation is to use configdrive which allows you to execute commands post server creation, as well as to install certain packages that are required.

The good thing about using this is you can get a server up and running with a single line of automation, and of course your configuration file (which contains all the automation). Here is the steps you need to do it, and it is actually really rather very simple!

Step 1. Create Automation File .cloud-config 

#cloud-config

packages:

 - apache2
 - php5
 - php5-mysql
 - mysql-server

runcmd:

 - wget http://wordpress.org/latest.tar.gz -P /tmp/
 - tar -zxf /tmp/latest.tar.gz -C /var/www/
 - mysql -e "create database wordpress; create user 'wpuser'@'localhost' identified by 'changemetoo'; grant all privileges on wordpress . \* to 'wpuser'@'localhost'; flush privileges;"
 - mysql -e "drop database test; drop user 'test'@'localhost'; flush privileges;"
 - mysqladmin -u root password 'changeme'

Install apache2, php5, php-mysql, mysqlserver, download wordpress to /tmp and then extract it into main /var/www folder. Create the wordpress database and user name.

Step 2: Create server using cloud-config in Supernova via the Rackspace API
(not hard! easy!)

supernova customer boot --config-drive=true --flavor performance1-1 --image 09de0a66-3156-48b4-90a5-1cf25a905207 --user-data cloud-config testing-configdrive


+--------------------------------------+-------------------------------------------------------------------------------+
| Property                             | Value                                                                         |
+--------------------------------------+-------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                        |
| OS-EXT-STS:power_state               | 0                                                                             |
| OS-EXT-STS:task_state                | scheduling                                                                    |
| OS-EXT-STS:vm_state                  | building                                                                      |
| RAX-PUBLIC-IP-ZONE-ID:publicIPZoneId |                                                                               |
| accessIPv4                           |                                                                               |
| accessIPv6                           |                                                                               |
| adminPass                            | SECUREPASSWORDHERE                                                            |
| config_drive                         | True                                                                          |
| created                              | 2015-10-20T11:10:23Z                                                          |
| flavor                               | 1 GB Performance (performance1-1)                                             |
| hostId                               |                                                                               |
| id                                   | ef084d0f-70cc-4366-b348-daf987909899                                          |
| image                                | Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM) (09de0a66-3156-48b4-90a5-1cf25a905207) |
| key_name                             | -                                                                             |
| metadata                             | {}                                                                            |
| name                                 | testing-configdrive                                                           |
| progress                             | 0                                                                             |
| status                               | BUILD                                                                         |
| tenant_id                            | 10000000                                                                      |
| updated                              | 2015-10-20T11:10:24Z                                                          |
| user_id                              | 05b18e859cad42bb9a5a35ad0a6fba2f                                              |
+--------------------------------------+-------------------------------------------------------------------------------+

In my case my supernova was setup already, however I have another article on how to setup supernova on this site, just take a look there for how to install it. MY supernova configuration looks like (with the API KEY removed ofcourse!)

[customer]
OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/
OS_AUTH_SYSTEM=rackspace
#OS_COMPUTE_API_VERSION=1.1
NOVA_RAX_AUTH=1
OS_REGION_NAME=LON
NOVA_SERVICE_NAME=cloudServersOpenStack
OS_PASSWORD=90bb3pd0a7MYMOCKAPIKEYc419572678abba136a2
OS_USERNAME=mycloudusername
OS_TENANT_NAME=100000

OS_TENANT_NAME is your customer number, take it from the url in mycloud.rackspace.com after logging on. OS_PASSWORD is your API KEY, get it from the account settings url in mycloud.rackspace.co.uk, and your OS_USERNAME, that is your username that you use to login to the Rackspace mycloud control panel. Simples!

Step 3: Confirm your server built as expected

root@testing-configdrive:~# ls /tmp
latest.tar.gz

root@testing-configdrive:~# ls /var/www/wordpress
index.php    readme.html      wp-admin            wp-comments-post.php  wp-content   wp-includes        wp-load.php   wp-mail.php      wp-signup.php     xmlrpc.php
license.txt  wp-activate.php  wp-blog-header.php  wp-config-sample.php  wp-cron.php  wp-links-opml.php  wp-login.php  wp-settings.php  wp-trackback.php

In my case, I noticed that everything went fine and ‘wordpress’ installed to /var/www just fine. But what if I wanted wordpress www dir configured to html by default? That’s pretty easy. It’s just an extra.

mv /var/www/html /var/www/html_old
mv /var/www/wordpress /var/www/html

So lets add that to our automation script:

#cloud-config

packages:

 - apache2
 - php5
 - php5-mysql
 - mysql-server

runcmd:

 - wget http://wordpress.org/latest.tar.gz -P /tmp/
 - tar -zxf /tmp/latest.tar.gz -C /var/www/; mv /var/www/html /var/www/html_old; mv /var/www/wordpress /var/www/html
 - mysql -e "create database wordpress; create user 'wpuser'@'localhost' identified by 'changemetoo'; grant all privileges on wordpress . \* to 'wpuser'@'localhost'; flush privileges;"
 - mysql -e "drop database test; drop user 'test'@'localhost'; flush privileges;"
 - mysqladmin -u root password 'changeme'

Job done. Just a case of re-running the command now:

supernova customer boot --config-drive=true --flavor performance1-1 --image 09de0a66-3156-48b4-90a5-1cf25a905207 --user-data cloud-config testing-configdrive

And then checking that our wordpress website loads correctly without any additional configuration or having to login to the machine! Not bad automation thar.

I could have quite easily achieved something like this by using the API directly. No supernova and no filesystem. Just the raw command! Yeah that’d be better than not bad!

Creating Post BUILD Automation thru API via CURL

Here’s how to do it.

Step 1. Prepare your execution script by converting it to BASE_64 character encoding

Unencoded Script:

#cloud-config

packages:

 - apache2
 - php5
 - php5-mysql
 - mysql-server

runcmd:

 - wget http://wordpress.org/latest.tar.gz -P /tmp/
 - tar -zxf /tmp/latest.tar.gz -C /var/www/; mv /var/www/html /var/www/html_old; mv /var/www/wordpress /var/www/html
 - mysql -e "create database wordpress; create user 'wpuser'@'localhost' identified by 'changemetoo'; grant all privileges on wordpress . \* to 'wpuser'@'localhost'; flush privileges;"
 - mysql -e "drop database test; drop user 'test'@'localhost'; flush privileges;"
 - mysqladmin -u root password 'changeme'

Encoded Script:

I2Nsb3VkLWNvbmZpZw0KDQpwYWNrYWdlczoNCg0KIC0gYXBhY2hlMg0KIC0gcGhwNQ0KIC0gcGhwNS1teXNxbA0KIC0gbXlzcWwtc2VydmVyDQoNCnJ1bmNtZDoNCg0KIC0gd2dldCBodHRwOi8vd29yZHByZXNzLm9yZy9sYXRlc3QudGFyLmd6IC1QIC90bXAvDQogLSB0YXIgLXp4ZiAvdG1wL2xhdGVzdC50YXIuZ3ogLUMgL3Zhci93d3cvIDsgbXYgL3Zhci93d3cvaHRtbCAvdmFyL3d3dy9odG1sX29sZDsgbXYgL3Zhci93d3cvd29yZHByZXNzIC92YXIvd3d3L2h0bWwNCiAtIG15c3FsIC1lICJjcmVhdGUgZGF0YWJhc2Ugd29yZHByZXNzOyBjcmVhdGUgdXNlciAnd3B1c2VyJ0AnbG9jYWxob3N0JyBpZGVudGlmaWVkIGJ5ICdjaGFuZ2VtZXRvbyc7IGdyYW50IGFsbCBwcml2aWxlZ2VzIG9uIHdvcmRwcmVzcyAuIFwqIHRvICd3cHVzZXInQCdsb2NhbGhvc3QnOyBmbHVzaCBwcml2aWxlZ2VzOyINCiAtIG15c3FsIC1lICJkcm9wIGRhdGFiYXNlIHRlc3Q7IGRyb3AgdXNlciAndGVzdCdAJ2xvY2FsaG9zdCc7IGZsdXNoIHByaXZpbGVnZXM7Ig0KIC0gbXlzcWxhZG1pbiAtdSByb290IHBhc3N3b3JkICdjaGFuZ2VtZSc=

Step 2: Get Authorization token from identity API endpoint

Command:


$ curl -s https://identity.api.rackspacecloud.com/v2.0/tokens -X 'POST'        -d '{"auth":{"passwordCredentials":{"username":"adambull", "password":"superBRAIN%!7912105!"}}}'        -H "Content-Type: application/json"

Response:

{"access":{"token":{"id":"AAD4gu67KlOPQeRSTJVC_8MLrTomBCxN6HdmVhlI4y9SiOa-h-Ytnlls2dAJo7wa60E9nQ9Se0uHxgJuHayVPEssmIm--MOCKTOKEN_EXAMPLE-0Wv5n0ZY0A","expires":"2015-10-21T15:06:44.577Z"

It’s also possible to use your API Key to retrieve the TOKEN ID used by API:
(if you don’t like using your control panel password!)

curl -s https://identity.api.rackspacecloud.com/v2.0/tokens -X 'POST' \
       -d '{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"yourUserName", "apiKey":"yourApiKey"}}}' \
       -H "Content-Type: application/json" | python -m json.tool

Step 3: Construct Script to Execute Command directly thru API


#!/bin/sh

# Your Rackspace ACCOUNT DDI, look for a number like below when you login to the Rackspace mycloud controlpanel
account='10000000'

# Using the token that was returned to us in step 2
token="AAD4gu6FH-KoLCKiPWpqHONkCqGJ0YiDuO6yvQG4J1jRSjcQoZSqRK94u0jaYv5BMOCKTOKENpMsI3NEkjNqApipi0Lr2MFLjw"

# London Datacentre Endpoint, could by SYD, IAD, ORD, DFW etc
curl -v https://lon.servers.api.rackspacecloud.com/v2/$account/servers \
       -X POST \
       -H "X-Auth-Project-Id: $account" \
       -H "Content-Type: application/json" \
       -H "Accept: application/json" \
       -H "X-Auth-Token: $token" \
       -d '{"server": {"name": "testing-cloud-init-api", "imageRef": "09de0a66-3156-48b4-90a5-1cf25a905207", "flavorRef": "general1-1", "config_drive": "true", "user_data": "I2Nsb3VkLWNvbmZpZw0KDQpwYWNrYWdlczoNCg0KIC0gYXBhY2hlMg0KIC0gcGhwNQ0KIC0gcGhwNS1teXNxbA0KIC0gbXlzcWwtc2VydmVyDQoNCnJ1bmNtZDoNCg0KIC0gd2dldCBodHRwOi8vd29yZHByZXNzLm9yZy9sYXRlc3QudGFyLmd6IC1QIC90bXAvDQogLSB0YXIgLXp4ZiAvdG1wL2xhdGVzdC50YXIuZ3ogLUMgL3Zhci93d3cvIDsgbXYgL3Zhci93d3cvaHRtbCAvdmFyL3d3dy9odG1sX29sZDsgbXYgL3Zhci93d3cvd29yZHByZXNzIC92YXIvd3d3L2h0bWwNCiAtIG15c3FsIC1lICJjcmVhdGUgZGF0YWJhc2Ugd29yZHByZXNzOyBjcmVhdGUgdXNlciAnd3B1c2VyJ0AnbG9jYWxob3N0JyBpZGVudGlmaWVkIGJ5ICdjaGFuZ2VtZXRvbyc7IGdyYW50IGFsbCBwcml2aWxlZ2VzIG9uIHdvcmRwcmVzcyAuIFwqIHRvICd3cHVzZXInQCdsb2NhbGhvc3QnOyBmbHVzaCBwcml2aWxlZ2VzOyINCiAtIG15c3FsIC1lICJkcm9wIGRhdGFiYXNlIHRlc3Q7IGRyb3AgdXNlciAndGVzdCdAJ2xvY2FsaG9zdCc7IGZsdXNoIHByaXZpbGVnZXM7Ig0KIC0gbXlzcWxhZG1pbiAtdSByb290IHBhc3N3b3JkICdjaGFuZ2VtZSc="}}' \
      | python -m json.tool

Zomg what does this mean?

X-Auth-Token: is just the header that is sent to authorise your request. You got the token using your mycloud username and password, or mycloud username and API key in step 2.
ImageRef: this is just the ID assigned to the base image of Ubuntu LTS 14.04. Take a look below at all the different images you can use (and the image id of each):

$ supernova customer image-list

| ade87903-9d82-4584-9cc1-204870011de0 | Arch 2015.7 (PVHVM)                                          | ACTIVE |                                      |
| fdaf64c7-d9f3-446c-bd7c-70349305ae91 | CentOS 5 (PV)                                                | ACTIVE |                                      |
| 21612eaf-a350-4047-b06f-6bb8a8a7bd99 | CentOS 6 (PV)                                                | ACTIVE |                                      |
| fabe045f-43f8-4991-9e6c-5cabd617538c | CentOS 6 (PVHVM)                                             | ACTIVE |                                      |
| 6595f1b7-e825-4bd2-addc-c7b1c803a37f | CentOS 7 (PVHVM)                                             | ACTIVE |                                      |
| 2c12f6da-8540-40bc-b974-9a72040173e0 | CoreOS (Alpha)                                               | ACTIVE |                                      |
| 8dc7d5d8-4ad4-41b6-acf1-958dfeadcb17 | CoreOS (Beta)                                                | ACTIVE |                                      |
| 415ca2e6-df92-44e6-ba95-8ee36b436b24 | CoreOS (Stable)                                              | ACTIVE |                                      |
| eaaf94d8-55a6-4bfa-b0a8-473febb012dc | Debian 7 (Wheezy) (PVHVM)                                    | ACTIVE |                                      |
| c3aacaf9-8d1e-4d41-bb47-045fbc392a1c | Debian 8 (Jessie) (PVHVM)                                    | ACTIVE |                                      |
| 081a8b12-515c-41c9-8ce4-13139e1904f7 | Debian Testing (Stretch) (PVHVM)                             | ACTIVE |                                      |
| 498c59a0-3c26-4357-92c0-dd938baca3db | Debian Unstable (Sid) (PVHVM)                                | ACTIVE |                                      |
| 46975098-7799-4e72-8ae0-d6ef9d2d26a1 | Fedora 21 (PVHVM)                                            | ACTIVE |                                      |
| 0976b31e-f6d7-4d74-81e9-007fca25067e | Fedora 22 (PVHVM)                                            | ACTIVE |                                      |
| 7a1cf8de-7721-4d56-900b-1e65def2ada5 | FreeBSD 10 (PVHVM)                                           | ACTIVE |                                      |
| 7451d607-426d-416f-8d29-97e57f6f3ad5 | Gentoo 15.3 (PVHVM)                                          | ACTIVE |                                      |
| 79436148-753f-41b7-aee9-5acbde16582c | OpenSUSE 13.2 (PVHVM)                                        | ACTIVE |                                      |
| 05dd965d-84ce-451b-9ca1-83a134e523c3 | Red Hat Enterprise Linux 5 (PV)                              | ACTIVE |                                      |
| 783f71f4-d2d8-4d38-b2e1-8c916de79a38 | Red Hat Enterprise Linux 6 (PV)                              | ACTIVE |                                      |
| 5176fde9-e9d6-4611-9069-1eecd55df440 | Red Hat Enterprise Linux 6 (PVHVM)                           | ACTIVE |                                      |
| 92f8a8b8-6019-4c27-949b-cf9910b84ffb | Red Hat Enterprise Linux 7 (PVHVM)                           | ACTIVE |                                      |
| 36076d08-3e8b-4436-9253-7a8868e4f4d7 | Scientific Linux 6 (PVHVM)                                   | ACTIVE |                                      |
| 6118e449-3149-475f-bcbb-99d204cedd56 | Scientific Linux 7 (PVHVM)                                   | ACTIVE |                                      |
| 656e65f7-6441-46e8-978d-0d39beaaf559 | Ubuntu 12.04 LTS (Precise Pangolin) (PV)                     | ACTIVE |                                      |
| 973775ab-0653-4ef8-a571-7a2777787735 | Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM)                  | ACTIVE |                                      |
| 5ed162cc-b4eb-4371-b24a-a0ae73376c73 | Ubuntu 14.04 LTS (Trusty Tahr) (PV)                          | ACTIVE |                                      |
| ***09de0a66-3156-48b4-90a5-1cf25a905207*** | Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)                       | ACTIVE |                                      |
| 658a7d3b-4c58-4e29-b339-2509cca0de10 | Ubuntu 15.04 (Vivid Vervet) (PVHVM)                          | ACTIVE |                                      |
| faad95b7-396d-483e-b4ae-77afec7e7097 | Vyatta Network OS 6.7R9                                      | ACTIVE |                                      |
| ee71e392-12b0-4050-b097-8f75b4071831 | Windows Server 2008 R2 SP1                                   | ACTIVE |                                      |
| 5707f82f-43f0-41e0-8e51-bfb597852825 | Windows Server 2008 R2 SP1 + SQL Server 2008 R2 SP2 Standard | ACTIVE |                                      |
| b684e5a0-11a8-433e-a4b8-046137783e1b | Windows Server 2008 R2 SP1 + SQL Server 2008 R2 SP2 Web      | ACTIVE |                                      |
| d16fd3df-3b24-49ee-ae6a-317f450006e7 | Windows Server 2012                                          | ACTIVE |                                      |
| f495b41d-07e1-44c5-a3e8-65c4412a7eb8 | Windows Server 2012 + SQL Server 2012 SP1 Standard           | ACTIVE |                                      |

flavorRef: is simply referring to what server type to start up, it’s pretty darn simple 

$ supernova lon flavor-list

+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID               | Name                    | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+
| 2                | 512MB Standard Instance | 512       | 20   | 0         |      | 1     |             | N/A       |
| 3                | 1GB Standard Instance   | 1024      | 40   | 0         |      | 1     |             | N/A       |
| 4                | 2GB Standard Instance   | 2048      | 80   | 0         |      | 2     |             | N/A       |
| 5                | 4GB Standard Instance   | 4096      | 160  | 0         |      | 2     |             | N/A       |
| 6                | 8GB Standard Instance   | 8192      | 320  | 0         |      | 4     |             | N/A       |
| 7                | 15GB Standard Instance  | 15360     | 620  | 0         |      | 6     |             | N/A       |
| 8                | 30GB Standard Instance  | 30720     | 1200 | 0         |      | 8     |             | N/A       |
| compute1-15      | 15 GB Compute v1        | 15360     | 0    | 0         |      | 8     |             | N/A       |
| compute1-30      | 30 GB Compute v1        | 30720     | 0    | 0         |      | 16    |             | N/A       |
| compute1-4       | 3.75 GB Compute v1      | 3840      | 0    | 0         |      | 2     |             | N/A       |
| compute1-60      | 60 GB Compute v1        | 61440     | 0    | 0         |      | 32    |             | N/A       |
| compute1-8       | 7.5 GB Compute v1       | 7680      | 0    | 0         |      | 4     |             | N/A       |
| general1-1       | 1 GB General Purpose v1 | 1024      | 20   | 0         |      | 1     |             | N/A       |
| general1-2       | 2 GB General Purpose v1 | 2048      | 40   | 0         |      | 2     |             | N/A       |
| general1-4       | 4 GB General Purpose v1 | 4096      | 80   | 0         |      | 4     |             | N/A       |
| general1-8       | 8 GB General Purpose v1 | 8192      | 160  | 0         |      | 8     |             | N/A       |
| io1-120          | 120 GB I/O v1           | 122880    | 40   | 1200      |      | 32    |             | N/A       |
| io1-15           | 15 GB I/O v1            | 15360     | 40   | 150       |      | 4     |             | N/A       |
| io1-30           | 30 GB I/O v1            | 30720     | 40   | 300       |      | 8     |             | N/A       |
| io1-60           | 60 GB I/O v1            | 61440     | 40   | 600       |      | 16    |             | N/A       |
| io1-90           | 90 GB I/O v1            | 92160     | 40   | 900       |      | 24    |             | N/A       |
| memory1-120      | 120 GB Memory v1        | 122880    | 0    | 0         |      | 16    |             | N/A       |
| memory1-15       | 15 GB Memory v1         | 15360     | 0    | 0         |      | 2     |             | N/A       |
| memory1-240      | 240 GB Memory v1        | 245760    | 0    | 0         |      | 32    |             | N/A       |
| memory1-30       | 30 GB Memory v1         | 30720     | 0    | 0         |      | 4     |             | N/A       |
| memory1-60       | 60 GB Memory v1         | 61440     | 0    | 0         |      | 8     |             | N/A       |
| performance1-1   | 1 GB Performance        | 1024      | 20   | 0         |      | 1     |             | N/A       |
| performance1-2   | 2 GB Performance        | 2048      | 40   | 20        |      | 2     |             | N/A       |
| performance1-4   | 4 GB Performance        | 4096      | 40   | 40        |      | 4     |             | N/A       |
| performance1-8   | 8 GB Performance        | 8192      | 40   | 80        |      | 8     |             | N/A       |
| performance2-120 | 120 GB Performance      | 122880    | 40   | 1200      |      | 32    |             | N/A       |
| performance2-15  | 15 GB Performance       | 15360     | 40   | 150       |      | 4     |             | N/A       |
| performance2-30  | 30 GB Performance       | 30720     | 40   | 300       |      | 8     |             | N/A       |
| performance2-60  | 60 GB Performance       | 61440     | 40   | 600       |      | 16    |             | N/A       |
| performance2-90  | 90 GB Performance       | 92160     | 40   | 900       |      | 24    |             | N/A       |
+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+

Cloud Files Container Bulk Deletion Script written in BASH

Posted on by

So, we had a lot of customers asking for ways to delete all of their cloud files in a single container, instead of having to do manually. This is possible using the bulk delete function defined in the rackspace docs. Find below the steps required to do this.

Step 1: Make an auth.json file (for simplicity)

{
    "auth": {
        "RAX-KSKEY:apiKeyCredentials": {
            "username": "mycloudusername",
            "apiKey": "mycloudapikey"
        }
    }
}

It’s quite simple and nothing intimidating.

For step 2 I’m using an application called jq, to install it do


wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
mv jq-linux64 /bin/jq
alias jq = '/bin/jq'

Now you can use jq at the commandline.

Step 2: Set variable called $TOKEN that can store the api token password output, the nice thing is there is no token stored in the script so its kind of secure

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d @auth.json -H "Content-type: application/json" | jq .access.token.id | sed 's/"//g'`
echo $TOKEN

Step 3: Set a variable for the container name

# Container to Upload to
CONTAINER=meh2

Step 4: Populate a List of all the files in the $CONTAINER variable, in this case ‘meh2’. 

# Populate File List
echo "Populating File List"
curl -X GET https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh2 -H "X-Auth-Token: $TOKEN" > filelist.txt

Step 5: Add container name to the file listing by rewriting the output file filelist.txt to a deletelist.txt

sed -e "s/^/\/$CONTAINER\//" <  filelist.txt > deletelist.txt

Step 6: Bulk Delete Files thru API

echo "Deleting Files.."
curl -i -v -XDELETE -H"x-auth-token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567\?bulk-delete -T ./deletelist.txt

Step 7: Confirm the deletion success

# Confirm Deleted
echo "Confirming Deleted in $CONTAINER.."
curl -i -X GET https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh2 -H "X-Auth-Token: $TOKEN"

The completed script looks like this:


 Mass Delete Container

# Get Token

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d @auth.json -H "Content-type: application/json" | jq .access.token.id | sed 's/"//g'`
echo $TOKEN

# Container to Upload to
CONTAINER=meh2


# Populate File List
echo "Populating File List"
curl -X GET https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh2 -H "X-Auth-Token: $TOKEN" > filelist.txt


# Add Container Prefix
echo "Adding Container Prefix.."
sed -e "s/^/\/$CONTAINER\//" <  filelist.txt > deletelist.txt


# Delete Files
echo "Deleting Files.."
curl -i -v -XDELETE -H"x-auth-token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567\?bulk-delete -T ./deletelist.txt

# Confirm Deleted
echo "Confirming Deleted in $CONTAINER.."
curl -i -X GET https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh2 -H "X-Auth-Token: $TOKEN"

Pretty simple!

Running it.. 

* About to connect() to storage101.lon3.clouddrive.com port 443 (#0)

*   Trying 2a00:1a48:7900::100...

* Connected to storage101.lon3.clouddrive.com (2a00:1a48:7900::100) port 443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA

* Server certificate:

* 	subject: CN=storage101.lon3.clouddrive.com

* 	start date: May 18 00:00:00 2015 GMT

* 	expire date: Nov 17 23:59:59 2016 GMT

* 	common name: storage101.lon3.clouddrive.com

* 	issuer: CN=thawte DV SSL CA - G2,OU=Domain Validated SSL,O="thawte, Inc.",C=US

> DELETE /v1/MossoCloudFS_10045567?bulk-delete HTTP/1.1

> User-Agent: curl/7.29.0

> Host: storage101.lon3.clouddrive.com

> Accept: */*

> x-auth-token: AAA7uz-F91SDsaMOCKTOKEN-gOLeB5bbffh8GBGwAPl9F313Pcy4Xg_zP8jtgZolMOudXhsZh-nh9xjBbOfVssaSx_shBMqkxIEEgW1zt8xESJbZLIsvBTNzfVBlTitbUS4RarUOiXEw

> Content-Length: 515

> Expect: 100-continue

>

< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue

* We are completely uploaded and fine
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain
Content-Type: text/plain
< X-Trans-Id: tx010194ea9a104443b89bb-00161f7f1dlon3
X-Trans-Id: tx010194ea9a104443b89bb-001611f7f1dlon3
< Date: Thu, 15 Oct 2015 10:25:35 GMT
Date: Thu, 15 Oct 2015 10:25:35 GMT
< Transfer-Encoding: chunked
Transfer-Encoding: chunked

<
Number Deleted: 44
Number Not Found: 0
Response Body:
Response Status: 200 OK
Errors:
* Connection #0 to host storage101.lon3.clouddrive.com left intact

Restarting a Xen Server that is out of memory thru Hypervisor

Posted on by

So, you have a xen server, but the virtual machine is not responding, what do you do? You login to the hypervisor and fix it, all right!

Please note for sanitation that random strings have been used instead of real life UUID.

Step 1: Connect Hypervisor

ssh root@somehypervisoriporhostname

Step 2: Check Running Tasks (Task List)

[root@10-1-1-1 ~]# xe task-list
uuid ( RO)                : ff9ca1a3-fc29-a245-1f28-2adc646114a2
          name-label ( RO): Async.VM.clean_reboot
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.371


uuid ( RO)                : aff56852-6db4-1ab3-b2b1-33e48c797dbb
          name-label ( RO): Connection to VM console
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.000


[root@10-1-1-1 ~]# xe task-list params=all
uuid ( RO)                  : ff9ca1a3-fc29-a245-1f28-2adc646114a2
            name-label ( RO): Async.VM.clean_reboot
      name-description ( RO):
            subtask_of ( RO): 
              subtasks ( RO):
           resident-on ( RO): 43b6096b-09cd-4890-b51b-56e50de573ff
                status ( RO): pending
              progress ( RO): 0.372
                  type ( RO): 
                result ( RO):
               created ( RO): 20151014T15:01:17Z
              finished ( RO): 19700101T00:00:00Z
            error_info ( RO):
    allowed_operations ( RO): Cancel


uuid ( RO)                  : aff56852-6db4-1ab3-b2b1-33e48c797dbb
            name-label ( RO): Connection to VM console
      name-description ( RO):
            subtask_of ( RO): 
              subtasks ( RO):
           resident-on ( RO): 43b6096b-09cd-4890-b51b-56e50de573ff
                status ( RO): pending
              progress ( RO): 0.000
                  type ( RO): 
                result ( RO):
               created ( RO): 20151014T15:57:48Z
              finished ( RO): 19700101T00:00:00Z
            error_info ( RO):
    allowed_operations ( RO):

I could see that there were two tasks running on this slice:

[root@10-1-1-1 ~]# xe vm-list name-label=slice10011111
uuid ( RO)           : 4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff
     name-label ( RW): slice10011111
    power-state ( RO): running

This told me that the slice was running OK. So I am going to cancel the task pending for it


$ xe task-cancel uuid=ff9ca1a3-fc29-a245-1f28-2adc646114a2

Shutdown the server (HALT IT)


[root@10-1-1-1 ~]# xe vm-shutdown --force uuid=4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff
[root@10-1-1-1 ~]# xe vm-list name-label=slice10011111
uuid ( RO)           : 4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff
     name-label ( RW): slice10011111
    power-state ( RO): halted

Start the Virtual Machine

[root@10-1-1-1 ~]# xe vm-start uuid=4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff

At the end I wanted to check if the instance was still causing a large swap as it was when it was running out of memory! That is the reason why I had to start the server.

(echo "Slice IO_Read IO_Write Total"; (for uuid in $(xe vbd-list params=uuid | awk '$5{print $5}'); do xe vbd-param-list uuid=$uuid | grep -P "^\s*(io_|vm-name-label|vdi-name-label|vdi-uuid|device)" | awk '{if($1=="vdi-uuid") {hasswap="no";vdi_uuid=$4;}}{if($1=="vm-name-label") name=$4; if($1=="vdi-name-label") {if ($4 ~ /swap/) {hasswap="yes";name=name"-swap"}; if ($5 ~ /ephemeral/) name=name"-eph";} if($1=="device"){if($4=="hda" || $4=="xvda") name=name"-root"; if($4=="xvdc" && hasswap=="no") {vdicmd="xe vdi-list uuid="vdi_uuid" params=name-description --minimal | grep swap >> /dev/null"; swpname=system(vdicmd); if(swpname==0) name=name"-swap"};} if($1=="io_read_kbs") ioread=$4; if($1=="io_write_kbs") iowrite=$4}END{if(substr(name,0,9)!="XenServer") print name" "ioread" "iowrite" "ioread+iowrite}'; done) | sort -k4n) | column -t

Job done!

Creating Cloud Servers using php-OpenCloud & PHP

Posted on by

So, I thought after doing such a good job with Python that I would take my trouble to PHP. In this case I was running PHP at the commandline, but there is no reason you can’t use these in your web application. That’s one good thing about PHP, right there. It may be the only thing, but, it’s there!

Step 1. Setup php-opencloud and php and composer

yum install php-opencloud php composer

Step 2. Setup composer requirement for php-opencloud (this is what is required for the vendor/autoloader.php file)

composer require rackspace/php-opencloud

Step 3. Configure opencloud


require 'vendor/autoload.php';

use OpenCloud\Rackspace;

Step 4. Configure Authorisation Section, including my username, apikey and REGION ‘LON’. For Dallas Forth Worth this would be DFW, etc.

# Authentication

$client = new Rackspace(Rackspace::US_IDENTITY_ENDPOINT, array(
    'username' => 'myusername',
    'apiKey'   => '90ghaj4532asdsFgsdrghdi9832'
));

$service = $client->computeService(null, 'LON');

Step 5. Set Image to use to create server

$image = $service->image('d5bb9732-6468-4963-85b7-b6d1025cd0c7');

Step 6. Set Flavor to use to create server

$flavor = $service->flavor('general1-1');

Step 7. Proceed with server build

$server = $service->server();

$response = $server->create(array(
        'name'          =>      'Mein New Test Serven',
        'imageId'       =>      $image->getId(),
        'flavorId'      =>      $flavor->getId()
));
?>

Step 8. The completed php file will look something like :

?php

require 'vendor/autoload.php';


use OpenCloud\Rackspace;


# Authentication


$client = new Rackspace(Rackspace::US_IDENTITY_ENDPOINT, array(

    'username' => 'myusername',

    'apiKey'   => '90ghaj4532asdsFgsdrghdi9832'

));


$service = $client->computeService(null, 'LON');

#

# Cloud Image

$image = $service->image('d5bb9732-6468-4963-85b7-b6d1025cd0c7');

#

# Cloud Server Flavor

$flavor = $service->flavor('general1-1');


# Proceed with Server Build


$server = $service->server();


$response = $server->create(array(

        'name'          =>      'Mein New Test Serven',

        'imageId'       =>      $image->getId(),

        'flavorId'      =>      $flavor->getId()

));


?>

Creating Cloud Servers using Python & Pyrax

Posted on by

Today I have been playing around with some Python. This time using Pyrax and the Rackspace API. Here is how I did it. In my case I was using a CentOS 7 image.

Step 1. Install pip and pyrax

yum install python-pip gcc make
pip install --upgrade pip
pip install pyrax

Step 2. Consult the docs! 

 https://developer.rackspace.com/docs/cloud-servers/getting-started/

Which has support for JAVA, PHP, Python, GO, RUBY, .NET and more.

Step 3. Create your import directives

import os
import pyrax

Step 3. Create your Authorisation 

# Authentication Section
# myusername is the mycloud username for the Rackspace User in portal
# 99ghghghghgh12345a289872342 is the APIKEY, you need to replace these with the values you use for your account
pyrax.set_setting("identity_type", "rackspace")
pyrax.set_default_region('LON')
pyrax.set_credentials('myusername', '99ghghghghgh12345a289872342')

cs = pyrax.cloudservers

Step 4. Listing some flavours: It’s possible to list the different virtual machine flavors (HARDWARE TYPE)

flavor_list = cs.list_flavors()
print flavor_list

Step 5. Set the flavor we want to use to create a server. In this case we are spinning up a performance 1 server with 1GB RAM.

flavor = cs.flavors.get('performance1-1')

Step 6. Set the image we want to use to create the server. I have a custom image from a previous server I made I want to use.

image = pyrax.images.get('d9aa9583-6468-4963-85b7-b6d1025cd0c7')

Step 7. Create the server with the parameters: name, image and flavor.

server = cs.servers.create('testing1', image.id, flavor.id)

The complete file should look like:

import os
import pyrax

# Authentication Section

pyrax.set_setting("identity_type", "rackspace")
pyrax.set_default_region('LON')
pyrax.set_credentials('adambull', '99ghghghghgh12345a289872342')

cs = pyrax.cloudservers

flavor_list = cs.list_flavors()

flavor = cs.flavors.get('performance1-1')

image = pyrax.images.get('d9aa9583-6468-4963-85b7-b6d1025cd0c7')
server = cs.servers.create('testing1', image.id, flavor.id)

Using Rackspace Cloud Files & the API

Posted on by

Hi. So, when I started working at Rackspace I didn’t know very much about API. I know what it is, what it does, and why it’s used, but my experience was rather limited. So I was understandably a little bit concerned about using cloud files and API. Specifically using POST and GET thru CURL, and simple things such as authorization and identification thru header tokens.

First of all to use API, and make API requests we need an access token. The access token is totally different to my mycloud username, password and API Key itself. The token is a bit like a session, wheras the API key is a bit like a form username and password. It’s also worth noting, just like a http session, the $TOKEN will expire every now and then requiring you to authorise yourself to get a new token.

Authorisation & End Points

When you authorise yourself, you will get a token and a list of all the possible endpoints to GET and POST data, that is to say to retrieve or store records or query against particular search pattern, etc.

Step 1: Get User Token Thru Identity API using JSON auth structure

File: auth.json

{
    "auth": {
        "RAX-KSKEY:apiKeyCredentials": {
            "username": "mycloudusernamehere",
            "apiKey": "mycloudapikeyhere"
        }
    }
}

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d @auth.json -H "Content-type: application/json"

It is possible to do this without the auth.json file and just use the string, like so:

 curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"yourUserName","apiKey":"yourAPIPassword"}}}'  -H "Content-type: application/json" | python -m json.tool

It is also possible to connect to API using just USERNAME and API KEY:

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"passwordCredentials":{"username":"yourUserName","password":"yourmycloudpassword"}}}' -H "Content-type: application/json"

After running one of these commands you will receive a large, and something like the below will be inside it:

This includes your full ‘token’ ID. In this case ‘BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw’

This is the token we need to use to authenticate against the API.

 

       "token": {
            "RAX-AUTH:authenticatedBy": [
                "APIKEY"
            ],
            "expires": "2015-09-29T17:04:56.092Z",
            "id": "BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw",
            "tenant": {
                "id": "1000000",
                "name": "1000000"
            }
        },

Step 2. Uploading a File with CURL to a Cloud File Container

# Set the TOKEN Variable in the BASH SHELL
TOKEN='BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw'

# CURL REQUEST TO API
CURL -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/1.txt -T /Users/adam9261/1.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/2.txt -T /Users/adam9261/2.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/3.txt -T /Users/adam9261/3.txt -H "X-Auth-Token: $TOKEN"

Output


HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:45 GMT
Content-Length: 0
Etag: 8aec10927922e92a963f6a1155ccc773
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txca64c51affee434abaa79-0056097a34lon3
Date: Mon, 28 Sep 2015 17:34:45 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:46 GMT
Content-Length: 0
Etag: 83736eb7b9eb0dce9d11abcf711ca062
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx6df90d2dede54565bccfe-0056097a35lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:47 GMT
Content-Length: 0
Etag: 42f2826fff018420731e7bead0f124df
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx79612b28754c4bb3aedd6-0056097a36lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

In this case I had 3 txt files, each approximately 1MB, that I uploaded to cloud files using CURL and the commandline. The pertinent part is “X-Auth-Token: $TOKEN”, which is the header which contains my authorisation key which was set at the commandline previously with TOKEN= line above it.

Now, it’s possible using a manifest to append these 3 files together into a single file. For instance if these files were larger, say 5GB and at the maximum limit for a file, such as a large dvd-iso, to exceed the 5GB limit we’d need to split that large file up into smaller files. But we want the cloud files container to send the whole file, all 3 5GB parts as if it were one file, this can be achieved with manifests. Here is how to do it!

Creating a Manifest file, defining 3 file parts as one single file download in the data stream

Step 1: Create Manifest File
In my case I created a manifest file first which tells cloud files which individual files make up the large file

[

        {
                "path": "/meh/1.txt",
                "etag": "8aec10927922e92a963f6a1155ccc773",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/2.txt",
                "etag": "83736eb7b9eb0dce9d11abcf711ca062",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/3.txt",
                "etag": "42f2826fff018420731e7bead0f124df",
                "size_bytes": 1048587
        }
]

Step 2: Assign Manifest 


curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/mylargeappendedfile?multipart-manifest=put -d @manifest -H "X-Auth-Token: $TOKEN"

As you can see, it’s really quite simple, by giving the path, etag and size_bytes, it’s possible to append these 3 file references to a single file association. Now when downloading mylargeappendedfile from the cloud storage container, you’ll get /meh/1.txt /meh/2.txt and /meh/3.txt all appended to a single file. This is pretty handy when dealing with files over 5GB, as the maximum limit for an individual file is 5GB, but that doesn’t stop you from spreading some size across multiple files, just like for instance in a rar archive.

Generated Output:


HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:32:18 GMT
Content-Length: 0
Etag: "8f49c8861f0aef2eb750099223050c27"
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx64093603d9444e088f242-00560979a0lon3
Date: Mon, 28 Sep 2015 17:32:32 GMT