Using Rackspace Cloud Files & the API

Hi. So, when I started working at Rackspace I didn’t know very much about API. I know what it is, what it does, and why it’s used, but my experience was rather limited. So I was understandably a little bit concerned about using cloud files and API. Specifically using POST and GET thru CURL, and simple things such as authorization and identification thru header tokens.

First of all to use API, and make API requests we need an access token. The access token is totally different to my mycloud username, password and API Key itself. The token is a bit like a session, wheras the API key is a bit like a form username and password. It’s also worth noting, just like a http session, the $TOKEN will expire every now and then requiring you to authorise yourself to get a new token.

Authorisation & End Points

When you authorise yourself, you will get a token and a list of all the possible endpoints to GET and POST data, that is to say to retrieve or store records or query against particular search pattern, etc.

Step 1: Get User Token Thru Identity API using JSON auth structure

File: auth.json

{
    "auth": {
        "RAX-KSKEY:apiKeyCredentials": {
            "username": "mycloudusernamehere",
            "apiKey": "mycloudapikeyhere"
        }
    }
}

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d @auth.json -H "Content-type: application/json"

It is possible to do this without the auth.json file and just use the string, like so:

 curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"yourUserName","apiKey":"yourAPIPassword"}}}'  -H "Content-type: application/json" | python -m json.tool

It is also possible to connect to API using just USERNAME and API KEY:

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"passwordCredentials":{"username":"yourUserName","password":"yourmycloudpassword"}}}' -H "Content-type: application/json"

After running one of these commands you will receive a large, and something like the below will be inside it:

This includes your full ‘token’ ID. In this case ‘BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw’

This is the token we need to use to authenticate against the API.

 

       "token": {
            "RAX-AUTH:authenticatedBy": [
                "APIKEY"
            ],
            "expires": "2015-09-29T17:04:56.092Z",
            "id": "BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw",
            "tenant": {
                "id": "1000000",
                "name": "1000000"
            }
        },

Step 2. Uploading a File with CURL to a Cloud File Container

# Set the TOKEN Variable in the BASH SHELL
TOKEN='BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw'

# CURL REQUEST TO API
CURL -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/1.txt -T /Users/adam9261/1.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/2.txt -T /Users/adam9261/2.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/3.txt -T /Users/adam9261/3.txt -H "X-Auth-Token: $TOKEN"

Output


HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:45 GMT
Content-Length: 0
Etag: 8aec10927922e92a963f6a1155ccc773
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txca64c51affee434abaa79-0056097a34lon3
Date: Mon, 28 Sep 2015 17:34:45 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:46 GMT
Content-Length: 0
Etag: 83736eb7b9eb0dce9d11abcf711ca062
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx6df90d2dede54565bccfe-0056097a35lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:47 GMT
Content-Length: 0
Etag: 42f2826fff018420731e7bead0f124df
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx79612b28754c4bb3aedd6-0056097a36lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

In this case I had 3 txt files, each approximately 1MB, that I uploaded to cloud files using CURL and the commandline. The pertinent part is “X-Auth-Token: $TOKEN”, which is the header which contains my authorisation key which was set at the commandline previously with TOKEN= line above it.

Now, it’s possible using a manifest to append these 3 files together into a single file. For instance if these files were larger, say 5GB and at the maximum limit for a file, such as a large dvd-iso, to exceed the 5GB limit we’d need to split that large file up into smaller files. But we want the cloud files container to send the whole file, all 3 5GB parts as if it were one file, this can be achieved with manifests. Here is how to do it!

Creating a Manifest file, defining 3 file parts as one single file download in the data stream

Step 1: Create Manifest File
In my case I created a manifest file first which tells cloud files which individual files make up the large file

[

        {
                "path": "/meh/1.txt",
                "etag": "8aec10927922e92a963f6a1155ccc773",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/2.txt",
                "etag": "83736eb7b9eb0dce9d11abcf711ca062",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/3.txt",
                "etag": "42f2826fff018420731e7bead0f124df",
                "size_bytes": 1048587
        }
]

Step 2: Assign Manifest


curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/mylargeappendedfile?multipart-manifest=put -d @manifest -H "X-Auth-Token: $TOKEN"

As you can see, it’s really quite simple, by giving the path, etag and size_bytes, it’s possible to append these 3 file references to a single file association. Now when downloading mylargeappendedfile from the cloud storage container, you’ll get /meh/1.txt /meh/2.txt and /meh/3.txt all appended to a single file. This is pretty handy when dealing with files over 5GB, as the maximum limit for an individual file is 5GB, but that doesn’t stop you from spreading some size across multiple files, just like for instance in a rar archive.

Generated Output:


HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:32:18 GMT
Content-Length: 0
Etag: "8f49c8861f0aef2eb750099223050c27"
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx64093603d9444e088f242-00560979a0lon3
Date: Mon, 28 Sep 2015 17:32:32 GMT

haxed.me.uk

a place of infrastructure delights

Using Rackspace Cloud Files & the API

Authorisation & End Points

Creating a Manifest file, defining 3 file parts as one single file download in the data stream