Retrieve all CDN Rackcdn.com URL’s and URI

So, today we had a customer ask if I could trace down the containers for their specific domain CNAMES.

First I would dig the CNAME the customer setup. For instance cdn.customerdomain.com. This would give me a rackcdn.com link like;

# dig adam.haxed.me.uk

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.2 <<>> adam.haxed.me.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19402
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1500
;; QUESTION SECTION:
;adam.haxed.me.uk.		IN	A

;; ANSWER SECTION:
adam.haxed.me.uk.	3600	IN	CNAME	ceb47133a715104a5805-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.cf3.rackcdn.com.
ceb47133a715104a5805-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.cf3.rackcdn.com. 300 IN CNAME	a59.rackcdn.com.
a59.rackcdn.com.	281	IN	CNAME	a59.rackcdn.com.mdc.edgesuite.net.
a59.rackcdn.com.mdc.edgesuite.net. 300 IN CNAME	a61.dscg10.akamai.net.
a61.dscg10.akamai.net.	1	IN	A	104.86.110.99
a61.dscg10.akamai.net.	1	IN	A	104.86.110.115

;; Query time: 39 msec
;; SERVER: 83.138.151.81#53(83.138.151.81)
;; WHEN: Thu Apr 14 09:15:25 UTC 2016
;; MSG SIZE  rcvd: 261

This would give me detail of the CDN URL that my TLD points to. But, what if I am trying to track down the container, like my customer was? I will now create a script to list ALL rackCDN url's. Then we can search for the ceb47133 CNAME that adam.haxed.me.uk points to. This will give us a 'name' of the cloud files container that the rackcdn is associated/connected with.

USERNAME='mycloudusername'
APIKEY='mycloudapikey'

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

TENANT=10045567
API_ENDPOINT="https://cdn3.clouddrive.com/v1/MossoCloudFS_$TENANT"
#API_ENDPOINT="https://global.cdn.api.rackspacecloud.com/v1.0/$TENANT"
#API_ENDPOINT="https://cdn3.clouddrive.com/v1/MossoCloudFS_c2ad0d46-31e2-4c31-a60b-b611bb8e5f8b2"

curl -v -X GET $API_ENDPOINT/?format=json \
-H "X-Auth-Token: $TOKEN" | python -mjson.tool

It's well worth noting the API ENDPOINT is different from customer to customer so you may wish to retrieve all of the endpoints to check you have the right CDN endpoint. See at the bottom of the page how to check your endpoint is correct if you get permission error, it is likely the API endpoint. It's different for each customer I have learnt.

[
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://a30ae7cddb38b2112bce-03b08b0e5c91ea60f938585ef20a12d7.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://1627826b1dc042d6b3be-03b08b0e5c91ea60f938585ef20a12d7.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://ee7e9298372b91eea2d2-03b08b0e5c91ea60f938585ef20a12d7.r91.stream.cf3.rackcdn.com",
        "cdn_uri": "http://beb2ec8d649b0d717ef9-03b08b0e5c91ea60f938585ef20a12d7.r91.cf3.rackcdn.com",
        "log_retention": false,
        "name": "some.com.cdn.container",
        "ttl": 86400
    },
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://0381268aadeda8ceab1e-37d5bb63c6aad292ad490c7fddb2f62f.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://5b190eda013130300b94-37d5bb63c6aad292ad490c7fddb2f62f.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://5f756e93360bbef82e84-37d5bb63c6aad292ad490c7fddb2f62f.r75.stream.cf3.rackcdn.com",
        "cdn_uri": "http://47aabb1759520adb10a1-37d5bb63c6aad292ad490c7fddb2f62f.r75.cf3.rackcdn.com",
        "log_retention": false,
        "name": "container-001",
        "ttl": 604800
    },
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://006acc500edc34a84075-1257f240203d0254bc8c5602aafda48d.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://b68de0566314da76870d-1257f240203d0254bc8c5602aafda48d.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://632bed500bfc691eb677-1257f240203d0254bc8c5602aafda48d.r49.stream.cf3.rackcdn.com",
        "cdn_uri": "http://b52a6ade17a64c459d85-1257f240203d0254bc8c5602aafda48d.r49.cf3.rackcdn.com",
        "log_retention": false,
        "name": "container-002",
        "ttl": 604800
    },
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://38d59ebf089e8ebe00a0-6490a1e5c1b40c9f5aaee7a62e1812f7.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://02a84412d877be1b8313-6490a1e5c1b40c9f5aaee7a62e1812f7.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://b8b8fe52062f7fb25f43-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.stream.cf3.rackcdn.com",
        "cdn_uri": "http://ceb47133a715104a5805-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.cf3.rackcdn.com",
        "log_retention": false,
        "name": "scripts",
        "ttl": 259200
    },
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://0c29cc67d5299ac41fa0-1426fb5304d7a905cdef320e9b667254.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://4df79706147258ab315b-1426fb5304d7a905cdef320e9b667254.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://66baf30a268d99e66228-1426fb5304d7a905cdef320e9b667254.r68.stream.cf3.rackcdn.com",
        "cdn_uri": "http://8b27955f0b728515adde-1426fb5304d7a905cdef320e9b667254.r68.cf3.rackcdn.com",
        "log_retention": false,
        "name": "test",
        "ttl": 259200
    },
    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://cc1d82abf0fbfced78b7-53ad0106578d82de3911abdf4b56c326.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://7173244627f44933cf9e-53ad0106578d82de3911abdf4b56c326.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://dd74f1300c187bb447f3-53ad0106578d82de3911abdf4b56c326.r30.stream.cf3.rackcdn.com",
        "cdn_uri": "http://cb7b587bb6e7186c9308-53ad0106578d82de3911abdf4b56c326.r30.cf3.rackcdn.com",
        "log_retention": false,
        "name": "test2",
        "ttl": 259200
    }
]
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool`

As we can see below the ceb46 rackcdn link is the 'scripts' container. the CNAME adam.haxed.me.uk points to the rackcdn.com domain http://ceb47133a715104a5805-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.cf3.rackcdn.com which is 'pointing' at the cloud files 'scripts' folder.

    {
        "cdn_enabled": true,
        "cdn_ios_uri": "http://38d59ebf089e8ebe00a0-6490a1e5c1b40c9f5aaee7a62e1812f7.iosr.cf3.rackcdn.com",
        "cdn_ssl_uri": "https://02a84412d877be1b8313-6490a1e5c1b40c9f5aaee7a62e1812f7.ssl.cf3.rackcdn.com",
        "cdn_streaming_uri": "http://b8b8fe52062f7fb25f43-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.stream.cf3.rackcdn.com",
        "cdn_uri": "http://ceb47133a715104a5805-6490a1e5c1b40c9f5aaee7a62e1812f7.r59.cf3.rackcdn.com",
        "log_retention": false,
        "name": "scripts",
        "ttl": 259200
    },

Simple enough!

Download Rackspace Cloud Files Containers (all of them) for archiving

curl -sS https://getcomposer.org/installer | php
php composer.phar require rackspace/php-opencloud

< ? php
/**
 *  "require": {
 *      "rackspace/php-opencloud": "dev-master"
 *  }
 */

ini_set('memory_limit', '2048M'); // size must be bigger than the biggest file
ini_set('max_execution_time', 0);

require 'vendor/autoload.php';

use OpenCloud\Rackspace;

// Instantiate a Rackspace client.
$client = new Rackspace(Rackspace::US_IDENTITY_ENDPOINT, array(
    'username' => 'mycloudusernamehere',
    'apiKey'   => 'myapikeyhere'
));

$objStore = $client->objectStoreService('cloudFiles', 'LON');
$savePath = __DIR__.'/archive/files/';

// get our containers
print("Containers:\n");
$containersList = $objStore->listContainers();

while($container = $containersList->Next()) {
    if( ! in_array($container->name, array('.CDN_ACCESS_LOGS', ''))) {
        printf("*** %s\n", $container->name);

        $containerDir = $savePath.$container->name.'/';
        if (!is_dir($containerDir)) {
            mkdir($containerDir, 0777, true);
        }

        $files = $container->ObjectList();
        while($o = $files->Next()) {
            $file_name = $o->getName();

            if (file_exists($containerDir . $file_name)) {
                echo '## '.$containerDir.$file_name.' already exists'."\n";
                continue;
            }

            // Get our object
            $file = $container->getObject($file_name);

            if (strpos($file->getName(), '') !== false) {
                continue;
            }

            $tempDir = $containerDir . dirname($file->getName()) . '/';
            if (!is_dir($tempDir)) {
                mkdir($tempDir, 0777, true);
            }

            if (file_put_contents($containerDir . $file_name, $file->getContent())) {
                printf("** %s - OK\n", $file->getName());
            } else {
                printf("** %s - KO\n", $file->getName());
            }
            unset($file);
        }
    }
}

The script can be executed like

php script.php 

php cloud-files-php-downloader2.php
Containers:
*** meine.org.cdn.container
** wp-content/themes/twentyfifteen/css/editor-style.css - OK
** wp-content/themes/twentyfifteen/css/ie.css - OK
** wp-content/themes/twentyfifteen/css/ie7.css - OK
** wp-content/themes/twentyfifteen/genericons/Genericons.ttf - OK

Installing and using Swiftly with Rackspace Cloud Files

So you want a commandline tool for managing cloud files containers? Enter, swiftly. Now you’ve seen examples before with curl, fog and pyrax, but here’s a special one which is a commandline application.

# Upgrade pip just for hoots
pip install --upgrade pip

# Install swiftly
pip install swiftly

Now swiftly is installed, how about configuring it in your home directory

# put a .swiftly.conf in your home directory, if your root i.e. /root/.swiftly.
vi ~/.swiftly.conf

This is what the configuration should look like

[swiftly]                                                                       
auth_user = mycloudusername                                                        
auth_key = mycloudapikey                                  
auth_url = https://identity.api.rackspacecloud.com/v2.0                         
region = lon
bash-4.2 Tue Feb 16 15:04:34 pirax-test ~# swiftly get

current
export
exports
gallery
images
lb_153727_TESTING_Nov_2015
lb_153727_TESTING_Oct_2015
magento-stack-single-magento_server-gxlxbm7b6be2
meh
meh2
rackspace_orchestration_templates_store
scripts
testing
testing999
versions

This gives us the output of all of the cloud containers as shown above. Pretty cool. But what about placing files in a container?

swiftly put -i ~/myfile.txt CONTAINER/path/to/file/somefilenamethatsdifferent.txt

So If I wanted to upload to meh2 I would do an

swiftly put -i ~/mylocalfile.txt meh2/some/container/path/somefileiuploaded.txt

The destination file can be called mylocalfile.txt if you want but I want to illustrate the target name can be different to the local source name.

Stopping a process from crapping out without PIDcrap

So, this one comes up a lot too. So you wanna run a process, and you don’t want it to crap out, you don’t want PIDCrap or any other lunatic solution that simply doesn’t work 100%. Well, welcome to until.

I’ve been executing a ruby script that does some stuff with fog.

ruby my-fog-cloud-files-container-deleter-thingy.rb

but, it keeps crapping out with lots of errors

I figured crapout no more and nabbed this handy snippet, credit to good ole stackoverflow

until ruby my-fog-cloud-files-container-deleter-thingy.rb; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

Now when it craps out, it continues where it left off.. nice, simple, elegant.

I don’t know what kind of error handling swiftly and pyrax has available in built, but this is a nice way to do it. Theoretically this oneliner might be of use for turbolift as well as any other batch like job which might end prematurely before the code deploy finishes. I wish cloud init had something like this

Removing and purging all Rackspace Cloud Files from all containers

So, this one comes up a lot at work, and thanks to knowitnot.com I came accross this example using ruby and fog, its simple, dont runnaway!

Regex lets you define the exact things you want to delete, but be careful. I’m using .* , I’ve removed it and replaced it with some_regex, to make sure people don’t accidentally nuke their cloud files 😀

Install first

yum install rubygem-fog.noarch
#!/usr/bin/env ruby
# Author: Jason Barnett 

require 'fog'

def delete_file(container, file_num, max_tries)
  max_retries ||= max_tries
  try = 0
  puts "(#{file_num} of #{container.files.count}) Removing #{container.files[file_num].key}"
  begin
    container.files[file_num].destroy
  rescue Excon::Errors::NotFound, Excon::Errors::Timeout, Fog::Storage::Rackspace::NotFound => e
    if try == max_retries
      $stderr.puts e.message
    else
      try += 1
      puts "Retry \##{try}"
      retry
    end
  end
end

def equal_div(first, last, num_of_groups)
  total      = last - first
  group_size = total / num_of_groups + 1

  top    = first
  bottom = top + group_size
  blocks = 1.upto(num_of_groups).inject([]) do |result, x|
    bottom = last if bottom > last
    result << [ top, bottom ]

    top    += group_size + 1
    bottom =  top + group_size

    result
  end

  blocks
end

service = Fog::Storage.new({
    :provider             => 'Rackspace',               # Rackspace Fog provider
    :rackspace_username   => 'your_rackspace_username', # Your Rackspace Username
    :rackspace_api_key    => 'your_api_key',            # Your Rackspace API key
    :rackspace_region     => :ord,                      # Defaults to :dfw
    :connection_options   => {},                        # Optional
    :rackspace_servicenet => false                      # Optional, only use if you're the Rackspace Region Data Center
})

containers = service.directories.select do |s|
  s.key =~ /^some_regex/  # Only delete containers that match the regexp
end

TOT_THREADS = 4
threads     = []

containers.each do |container|
  puts
  puts "-----------------------------------------"
  puts "-- Removing _ALL_ objects from #{container.key}"
  puts "-----------------------------------------"
  puts

  #puts "container.files.count: #{container.files.count}"

  ## separates the number of files into equal groups to distribute to each thread
  mygroups = equal_div(0, container.files.count - 1, TOT_THREADS)

  0.upto(TOT_THREADS - 1) do |thread|
    threads << Thread.new([ container, mygroups[thread] ]) { |tObject|
      tObject[1][0].upto(tObject[1][1]) do |x|
        delete_file(tObject[0], x, 5)
      end
    }

  end
  threads.each { |aThread|  aThread.join }
  puts "Deleting #{container.key}"
  container.destroy
end

The script works, supporting multiple threads:

(8178 of 10000) Removing conv/2015/04/15/08/bdce78e4cab875e9e17eeeae051b1128.log.gz
(3183 of 10000) Removing conv/2015/03/17/18/0ea71bf7a834fc02b01c7f65c4eb23b0.log.gz
(5685 of 10000) Removing conv/2015/04/01/08/e4d6a7a2ee83d0be116cca6b1a92ad2a.log.gz
(682 of 10000) Removing conv/2015/02/26/07/163f33cf4e33c64139ab3bd7092a9478.log.gz
(8179 of 10000) Removing conv/2015/04/15/09/09ef6c4ecaa9341e76648b6e175db888.log.gz
(5686 of 10000) Removing conv/2015/04/01/09/23fa1dce145c8343efd8e0227fd41e35.log.gz
(3184 of 10000) Removing conv/2015/03/17/18/66cb5ac707c40777a1a78b1486e1c4f3.log.gz
(683 of 10000) Removing conv/2015/02/26/07/35f28368ed45b2fb7c7076c3b78eb008.log.gz
(5687 of 10000) Removing conv/2015/04/01/09/42882b705c2d7cfe5c726ddec3e457fc.log.gz
(3185 of 10000) Removing conv/2015/03/17/18/73dbb421db0093279138b6f69f246c06.log.gz
(8180 of 10000) Removing conv/2015/04/15/09/1a01e6b1779b76b5221b1c8c08398b00.log.gz
(684 of 10000) Removing conv/2015/02/26/07/44a59e91f6632383f18ff253e4404dba.log.gz
(3186 of 10000) Removing conv/2015/03/17/18/8f8ba5791ab0c1979b168517b229ef74.log.gz
(5688 of 10000) Removing conv/2015/04/01/09/8ff43c8a5de8a41ee9890686150c8c19.log.gz
(8181 of 10000) Removing conv/2015/04/15/09/28035bc7882fe36bb33da670e6c71ac0.log.gz
(685 of 10000) Removing conv/2015/02/26/07/7947d1b3f712d175281d33a6073531d9.log.gz
(8182 of 10000) Removing conv/2015/04/15/09/34a597377f63709a75c9fd6e8a68e073.log.gz
(3187 of 10000) Removing conv/2015/03/17/18/a92a58f166cc9bd1ff54a5f996bb776b.log.gz
(5689 of 10000) Removing conv/2015/04/01/09/9a728570208bcac32fd11f1581a57fbb.log.gz

Testing CDN Consistency with bash date time curl while loop

This is a simple one. Soa customer was complaining that after 3 minutes the cache time of the file on his CDN was changing. I wanted to built a way to test the consistency of the requests. Here is how I did it.

file curl-format.txt

     timenamelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n

Short, simple, and to the point;

while ((1!=0)); do date; curl -w "@curl-format.txt" -o /dev/null -s "https://www.somecdndomain.secure.raxcdn.com/img/upload/3someimage_t32337827238.jpg"; done;

Output looks like:

                    ----------
         time_total:  0.395
Tue Feb  9 09:03:28 UTC 2016
      time_namelookup:  0.151
       time_connect:  0.154
    time_appconnect:  0.332
   time_pretransfer:  0.333
      time_redirect:  0.000
 time_starttransfer:  0.338
                    ----------
         time_total:  0.351
Tue Feb  9 09:03:28 UTC 2016
      time_namelookup:  0.151
       time_connect:  0.154
    time_appconnect:  0.324
   time_pretransfer:  0.324
      time_redirect:  0.000
 time_starttransfer:  0.331
                    ----------
         time_total:  0.347
Tue Feb  9 09:03:29 UTC 2016
      time_namelookup:  0.151
       time_connect:  0.154
    time_appconnect:  0.385
   time_pretransfer:  0.385
      time_redirect:  0.000
 time_starttransfer:  0.391
                    ----------
         time_total:  0.404
Tue Feb  9 09:03:29 UTC 2016
      time_namelookup:  0.151
       time_connect:  0.155
    time_appconnect:  0.348
   time_pretransfer:  0.349
      time_redirect:  0.000
 time_starttransfer:  0.357
                    ----------
         time_total:  0.374
Tue Feb  9 09:03:30 UTC 2016
      time_namelookup:  0.151
       time_connect:  0.155
    time_appconnect:  0.408
   time_pretransfer:  0.409
      time_redirect:  0.000
 time_starttransfer:  0.417
                    ----------
         time_total:  0.433
Tue Feb  9 09:03:30 UTC 2016

pretty handy andy.

With headers

# while ((1!=0)); do date; curl -IL -w "@curl-format.txt" -s "https://www.scdn3.secure.raxcdn.com/img/upload/3_sdsdsds6a9e80df0baa19863ffb8.jpg"; sleep 180; done;

Using Cloud Files Versioning, Setting up from Scratch

Sooooo.. you want to use cloud-files, but, you want to have versioning? no problem! Here’s how you do it from the ground up.

Authorise yourself thru identity API

Basically… set the token by querying the identity api with username and password..

!/bin/bash

# Username used to login to control panel
USERNAME='mycloudusername'

# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY='mycloudapikey'

# This section simply retrieves the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

If you were to add to this file;

echo $TOKEN

You’d see this when running it

# ./versioning.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5143  100  5028  100   115   3991     91  0:00:01  0:00:01 --:--:--  3996
8934534DFGJdfSdsdFDS232342DFFsDDFIKJDFijTx8WMIDO8CYzbhyViGGyekRYvtw3skCYMaqIWhw8adskfjds894FGKJDFKj34i2jgidgjdf@DFsSDsd

To understand how the curl works to authorise itself with the identity API and specifically how the TOKEN is extracted from the return output and set in the script, here is the -v verbose output


# ./versioning.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* About to connect() to identity.api.rackspacecloud.com port 443 (#0)
*   Trying 72.3.138.129...
* Connected to identity.api.rackspacecloud.com (72.3.138.129) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* 	subject: CN=identity.api.rackspacecloud.com,OU=Domain Validated,OU=Thawte SSL123 certificate,OU=Go to https://www.thawte.com/repository/index.html,O=identity.api.rackspacecloud.com
* 	start date: Nov 14 00:00:00 2011 GMT
* 	expire date: Nov 12 23:59:59 2016 GMT
* 	common name: identity.api.rackspacecloud.com
* 	issuer: CN=Thawte DV SSL CA,OU=Domain Validated SSL,O="Thawte, Inc.",C=US
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> POST /v2.0/tokens HTTP/1.1
> User-Agent: curl/7.29.0
> Host: identity.api.rackspacecloud.com
> Accept: */*
> Content-type: application/json
> Content-Length: 115
>
} [data not shown]
* upload completely sent off: 115 out of 115 bytes
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 02 Feb 2016 18:19:06 GMT
< Content-Type: application/json
< Content-Length: 5028
< Connection: keep-alive
< X-NewRelic-App-Data: Censored
< vary: Accept, Accept-Encoding, X-Auth-Token
< Front-End-Https: on
<
{ [data not shown]
100  5143  100  5028  100   115   3825     87  0:00:01  0:00:01 --:--:--  3826
* Connection #0 to host identity.api.rackspacecloud.com left intact
{
    "access": {
        "serviceCatalog": [
            {
                "endpoints": [
                    {
                        "internalURL": "https://snet-storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567",
                        "publicURL": "https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567",
                        "region": "LON",
                        "tenantId": "MossoCloudFS_10010101"
                    }
                ],
                "name": "cloudFiles",
                "type": "object-store"
            },
            {
   "token": {
            "RAX-AUTH:authenticatedBy": [
                "APIKEY"
            ],
            "expires": "2016-02-03T18:31:18.838Z",
            "id": "#$dfgkldfkl34klDFGDFGLK#$OFDOKGDFODJ#$OFDOGIDFOGI34ldfldfgkdo34lfdFGDKDFGDODFKDFGDFLK",
            "tenant": {
                "id": "10010101",
                "name": "10010101"
            }
        },

This is truncated, the output is larger, but basically the "token" section is stripped away at the id: part so that only the string is left, then that password is added into the TOKEN variable.

So no you understand auth.

Create the Version container

This contains all of the version changes of any file

i.e. if you overwrite a file 10 times, all 10 versions will be saved

# Create Versioning Container (Backup versions)
curl -i -XPUT -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions

Note we use $TOKEN , which is basically just hte password with the X-Auth-Token Header. -H means 'send this header'. X-Auth-Token is the header name, and $TOKEN is the password we populated in the variable in the first auth section above.

Create a Current Container

This only contains the 'current' or most latest version of the file

# Create current container (latest versions)
curl -i -XPUT -H "X-Auth-Token: $TOKEN" -H  "X-Versions-Location: versions"  https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current

I'm being a bit naughty here, I could make MossoCloudFS_10010101 a variable, like $CONTAINERSTORE or $CONTAINERPARENT. Or better $TENANTCONTAINER But meh. You get the idea. And learnt something.

Note importantly X-Versions-Location Header set when creating the 'current' cloud files container. It's asking to store versions of changes in current to the versions folder. Nice.

Create an object

Create the first version of an object, because its awesome

# Create an object
curl -i -XPUT --data-binary 1 -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current/myobject.obj

yay! My first object. I just put the number 1 in it. Not very imaginative but you get the idea. Now lets revise the object

Create a new version of the object

# Create a new version of the object (second version)
curl -i -XPUT --data-binary 2 -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current/myobject.obj

Create a list of the older versions of the object

# Create a list of the older versions of the object
curl -i -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions?prefix=008myobject.obj

Delete the current version of an object

# Delete the current version of the object
curl -i -XDELETE -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions?prefix=008myobject.obj

Pretty cool. Altogether now.

#!/bin/bash

# Username used to login to control panel
USERNAME='mycloudusername'

# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY=mycloudapikey'


# This section simply retrieves the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`


# This section requests the Glance API to copy the cloud server image uuid to a cloud files container called export

# Create Versioning Container (Backup versions)
curl -i -XPUT -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions

# Create current container (latest versions)
curl -i -XPUT -H "X-Auth-Token: $TOKEN" -H  "X-Versions-Location: versions"  https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current


# Create an object
curl -i -XPUT --data-binary 1 -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current/myobject.obj

# Create a new version of the object (second version)
curl -i -XPUT --data-binary 2 -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/current/myobject.obj

# Create a list of the older versions of the object
curl -i -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions?prefix=008myobject.obj

# Delete the current version of the object

curl -i -XDELETE -H "X-Auth-Token: $TOKEN" https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10010101/versions?prefix=008myobject.obj

What the output of the full script looks like:

# ./versioning.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5143  100  5028  100   115   4291     98  0:00:01  0:00:01 --:--:--  4290
HTTP/1.1 202 Accepted
Content-Length: 76
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx514bac5247924b5db247d-0056b0ecb7lon3
Date: Tue, 02 Feb 2016 17:51:51 GMT

Accepted

The request is accepted for processing.

HTTP/1.1 202 Accepted Content-Length: 76 Content-Type: text/html; charset=UTF-8 X-Trans-Id: tx7b7f42fc19b1428b97cfa-0056b0ecb8lon3 Date: Tue, 02 Feb 2016 17:51:52 GMT

Accepted

The request is accepted for processing.

HTTP/1.1 201 Created Last-Modified: Tue, 02 Feb 2016 17:51:53 GMT Content-Length: 0 Etag: c4ca4238a0b923820dcc509a6f75849b Content-Type: text/html; charset=UTF-8 X-Trans-Id: tx2495824253374261bf52a-0056b0ecb8lon3 Date: Tue, 02 Feb 2016 17:51:53 GMT HTTP/1.1 201 Created Last-Modified: Tue, 02 Feb 2016 17:51:54 GMT Content-Length: 0 Etag: c81e728d9d4c2f636f067f89cc14862c Content-Type: text/html; charset=UTF-8 X-Trans-Id: tx785e4a5b784243a1b8034-0056b0ecb9lon3 Date: Tue, 02 Feb 2016 17:51:54 GMT HTTP/1.1 204 No Content Content-Length: 0 X-Container-Object-Count: 2 Accept-Ranges: bytes X-Storage-Policy: Policy-0 X-Container-Bytes-Used: 2 X-Timestamp: 1454435183.80523 Content-Type: text/html; charset=UTF-8 X-Trans-Id: tx4782072371924905bc513-0056b0ecbalon3 Date: Tue, 02 Feb 2016 17:51:54 GMT

Rackspace Customer takes the time to improve my script :D

Wow. this was an awesome customer. Who was obviously capable in using the API but was struggling. So I thrown them my portable python -mjson parsing script for identity token and glance image export to cloud files. So,the customer wrote back, commenting that I’d made a mistake, specifically I had added ‘export’ instead of ‘exports’

#!/bin/bash

# Task ID - supply with command
TASK=$1
# Username used to login to control panel
USERNAME='myusername'
# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY='myapikeyhere'

# This section simply retrieves the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

# Requests progress of specified task
curl -X GET -H "X-Auth-Token: $TOKEN" "https://lon.images.api.rackspacecloud.com/v2/10010101/tasks/$TASK"

I just realised that the customer didn’t adapt the script to be able to pass in the image ID, on the initial export to cloud files.

Theoretically you could not only do the above but.. something like:

I just realised your script you sent checks the TASK. I just amended my initial script a bit further with your suggestion to accept myclouduser mycloudapikey and mycloudimageid

#!/bin/bash

# Username used to login to control panel
USERNAME=$1
# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY=$2

# Find the image ID you'd like to make available on cloud files
IMAGEID=$3

# This section simply retrieves the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`


# This section requests the Glance API to copy the cloud server image uuid to a cloud files container called export
curl https://lon.images.api.rackspacecloud.com/v2/10031542/tasks -X POST -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" -d '{"type": "export", "input": {"image_uuid": "'"$IMAGEID"'", "receiving_swift_container": "exports"}}'

# I thought that could theoretically process the output of the above and extract $TASK_ID to check the TASK too.

Note my script isn’t perfect but the customer did well!

This way you could simply provide to the script the cloud username, password and imageid. Then when the glance export starts, the task could be extracted in the same way as the TOKEN is from identity auth.

That way you could simply run something like

./myexportvhd.sh mycloudusername mycloudapikey mycloudimageid 

Not only would it start the image export to a set export folder.
But it'd provide you an update as to the task status.

You could go further, you could then watch the tasks status with a batch script while loop, until all show a complete or failed output and then record which ones succeeded and which ones failed. You could then create a batch script off that which downloaded and rsynched to somewhere the ones that succeeded.

Or..something like that.

I love it when one of our customers makes me think really hard. Gotta love that!

Testing Cloud Files API calls

So a customer was having issues with cloudfuse, the virtual ‘cloud files’ hard disk. So we needed to test whether their auth is working correctly:

#!/bin/bash
# Diagnostic script by Adam Bull
# Test Cloud Files Auth
# Tuesday, 02/02/2016

# Username used to login to control panel
USERNAME=’mycloudusername’

# Find the APIKey in the ‘account settings’ part of the menu of the control panel
APIKEY=’mycloudapikey’

# This section simply retrieves and sets the TOKEN

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d ‘{ “auth”:{“RAX-KSKEY:apiKeyCredentials”: { “username”:”‘$USERNAME'”, “apiKey”: “‘$APIKEY'” }} }’ -H “Content-type: application/json” | python -mjson.tool | grep -A5 token | grep id | cut -d ‘”‘ -f4`

# Container to Upload to (container must exist)

CONTAINER=testing

LOCALFILENAME=”/root/mytest.txt”
FILENAME=”mytest.txt”

# Put command, note MossoCloufFS_customeridhere needs to be populated with the correct value. This is the number in the mycloud url when logging in.
curl -i -v -X PUT “https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_101100/$CONTAINER/$FILENAME” -T “$LOCALFILENAME” -H “X-Auth-Token: $TOKEN”

Downloading exported Cloud Server Image from Cloud Files using BASH/curl/API

So, after succesfully exporting the image in the previous article, I wanted to download the VHD so I could use it on virtualbox at home.

#!/bin/bash

# Username used to login to control panel
USERNAME='adambull'
# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY='mycloudapikey'
# Find the image ID you'd like to make available on cloud files

# Simply replace mytenantidgoeshere10011111etc with just the account number, the number given in the url in mycloud control panel! replace everything after _ so it looks like _101110
TENANTID='MossoCloudFS_mytenantidgoeshereie1001111etc'

# This section simply retrieves the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

# Download the cloud files image

VHD_FILENAME=5fb64bf2-afae-4277-b8fa-0b69bc98185a.vhd
curl -o -i -X GET "https://storage101.lon3.clouddrive.com/v1/$TENANTID/exports/$VHD_FILENAME" \
-H "X-Auth-Token: $TOKEN"

Really really easy

Output looks like;

 ./download-image-id.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5143  100  5028  100   115   4470    102  0:00:01  0:00:01 --:--:--  4473
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1 3757M    1 38.1M    0     0  7231k      0  0:08:52  0:00:05  0:08:47 7875k