Using Rackspace Cloud Files, swiftly and cron to Backup Data to multiple data-centres cheaply

Posted on 17th May 2016 by azio

So, you have some really important data, so much so that 99.99% redundancy is not enough for you. One solution to this is to use multiple copies in multiple datacentres. Most enterprise backup will have on-site, an off-site, and an archival copy. What I’m going to show here is how to make 4 different copies of your data, in 4 different datacentres around the world. This will provide a very high redundancy of storage, and greatly reduce the likelihood of data loss. Although it costs a bit more, this kind of solution may be suitable for many small, medium and large businesses. Naturally, depending on the size of the data, and the importance of redundancy. You might not have many files to backup, perhaps a small cd worth.. it will be very inexpensive if you have a small backup to make. However, due to the way that cloud files is billed, copying data to cloud files costs money in bandwidth when writing from a server in London to a cloud files in Sydney, Chicago or Dallas for instance, so it’s very important to consider the impact of bandwidth costs when utilizing an additional 3 cloud files endpoints that are not in the local datacentre region. Which, is essentially what we are doing in this guide.

Setup swiftly

yum install python-devel python-pip -y
pip install swiftly eventlet

Create your swiftly environments (setting the name for each file)

==> /root/.swiftly-dfw.conf <==
[swiftly]
auth_user = myusername
auth_key = censored
auth_url = https://identity.api.rackspacecloud.com/v2.0
region = dfw

==> /root/.swiftly-iad.conf <==
[swiftly]
auth_user = myusername
auth_key = censored
auth_url = https://identity.api.rackspacecloud.com/v2.0
region = iad

==> /root/.swiftly-ord.conf <==
[swiftly]
auth_user = myusername
auth_key = censored
auth_url = https://identity.api.rackspacecloud.com/v2.0
region = ord

==> /root/.swiftly-syd.conf <==
[swiftly]
auth_user = myusername
auth_key = censored
auth_url = https://identity.api.rackspacecloud.com/v2.0
region = syd

Create your Script

# Adam Bull
# Adam Bull, Rackspace UK
# May 17, 2016


# This can be sequential or, it can be parallel, not sure which is better yet use & for parallel
# This backs up /documents file and puts it in the 'managed_backup' cloud files container at the following 4 datacentres ,DFW, IAD, ORD and SYD

swiftly --verbose --conf ~/.swiftly-dfw.conf --concurrency 100 put -i /documents /managed_backup
swiftly --verbose --no-snet --conf ~/.swiftly-iad.conf --concurrency 100 put -i /documents /managed_backup
swiftly --verbose --no-snet --conf ~/.swiftly-ord.conf --concurrency 100 put -i /documents /managed_backup
swiftly --verbose --no-snet --conf ~/.swiftly-syd.conf --concurrency 100 put -i /documents /managed_backup

Because the other 3 endpoints are in different datacentres, we can't use servicenet, so we defined --no-snet option for swiftly as above.

Execute your script

chmod +x multibackup.sh
./multibackup.sh

This obviously is a basic system and script of taking backups, and it is not for production use (yet). This is an alpha project I started today. The cool thing is that it works, and quite nicely. Although it is far from finished yet as a workable script.

Once the script is made, you can simply add it to crontab -e as you would usually. Make sure the user you execute with cron has access to the .conf files in their home directory!

Checking Load Balancer Connectivity & Automating it in some interesting ways

Posted on 16th May 2016 by azio

So, in a dream last night, I woke up realising I had forgot to write my automated load balancer connectivity checker.

Basically, sometimes a customer will complain their site is down because their ‘load balancer is broken’! In many cases, this is actually due to a firewall on one of the nodes behind the load balancer, or an issue with the webserver application listening on the port. So, I wrote a little piece of automation in the form of a BASH script, that accepts an Load Balancer ID and then uses the API to pull the server nodes behind that Load Balancer, including the ports being used to communicate, and then uses, either netcat or nmap to check that port for connectivity. There were a few ways to achieve this, but the below is what I was happiest with.

#!/bin/bash

# Username used to login to control panel
USERNAME='mycloudusernamegoeshere'

# Find the APIKey in the 'account settings' part of the menu of the control panel
APIKEY="apikeygoeshere"

# Your Rackspace account number (the number that is in the URL of the control panel after logging in)
ACCOUNT=100101010

# Your Rackspace loadbalancerID
LOADBALANCERID=157089

# Rackspace LoadBalancer Endpoint
ENDPOINT="https://lon.loadbalancers.api.rackspacecloud.com/v1.0"

# This section simply retrieves and sets the TOKEN
TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

#   (UNUSED) METHOD 1Extract IP addresses (Currently assuming port 80 only)
#curl -H "X-Auth-Token: $TOKEN" -H "Accept: application/json" -X GET "$ENDPOINT/$ACCOUNT/loadbalancers/$LOADBALANCERID/nodes" | jq .nodes[].address | xargs -i nmap -p 80 {}
#   (UNUSED) Extract ports
# curl -H "X-Auth-Token: $TOKEN" -H "Accept: application/json" -X GET "$ENDPOINT/$ACCOUNT/loadbalancers/$LOADBALANCERID/nodes" | jq .nodes[].port | xargs -i nmap -p 80 {}


# I opted for using this method to extract the important detail
curl -H "X-Auth-Token: $TOKEN" -H "Accept: application/json" -X GET "$ENDPOINT/$ACCOUNT/loadbalancers/$LOADBALANCERID/nodes" | jq .nodes[].address | sed 's/"//g' > address.txt
curl -H "X-Auth-Token: $TOKEN" -H "Accept: application/json" -X GET "$ENDPOINT/$ACCOUNT/loadbalancers/$LOADBALANCERID/nodes" | jq .nodes[].port > port.txt

# Loop thru both output files sequentially, order is important
# WARNING script does not ignore whitespace

while read addressfile1 <&3 && read portfile2 <&4; do
   ncat $addressfile1 $portfile2
done 3
Output looks a bit like;
# ./lbtest.sh

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  5143  100  5028  100   115   4731    108  0:00:01  0:00:01 --:--:--  4734

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   225  100   225    0     0    488      0 --:--:-- --:--:-- --:--:--   488

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   225  100   225    0     0    679      0 --:--:-- --:--:-- --:--:--   681

Ncat: No route to host.

Ncat: Connection timed out.
I plan to add some additional support that will check the load balancer is up, AND the servicenet connection between the cloud servers. 
Please note that this script must be run on a machine with access to servicenet network, in the same Rackspace Datacenter to be able to check servicenet connectivity of servers. The script can give false positives if strict firewall rules are setup on the cloud server nodes behind the load balancer. It's kind of alpha-draft but I thought I would share it as a proof of concept.
You will need to download and install jq to use it. To download jq please see; https://stedolan.github.io/jq/download/

Windows Password reset for Rackspace Cloud Servers

Posted on 6th May 2016 by azio

In the previous articles Using API and BASH to validate changing conditions and Reset windows administrator password using rescue mode without nova-agent I explained both the steps how to reset the password of a windows VM instance by modifying the SAM file by using a Linux ‘rescue’ image in the cloud, and, I also explained how to automate checks for BASH automation thru the API. The checks specifically waited until the server entered rescue, and then lifted the ipv4 address, connecting only when the rescue server had finished building.

That way the automation is handling the delay it takes, as well as setting and lifting the access credentials and ip address to use each time. Here is the complete script. Please note that backticks are deprecated but I’m a bit ‘oldskool’. This is a rough alpha, but it works really nicely. After testing it consistently allows ourselves, or our customers to reset a Windows Cloud Server password, in the case that a customer loses access to it, and cannot use other Rackspace services to do the reset. This effectively turns a useless server, back into a usable one again and saves a lot of time.

#!/bin/bash
# Adam Bull, Rackspace UK
# This script automates the resetting of windows passwords
# Arguments $1 == username
# Arguments $2 == apikey
# Arguments $3 == ddi
# Arguments $4 == instanceid

echo "Rackspace windows cloud server Password Reset"
echo "written by Adam Bull, Rackspace UK"
sleep 2
PASSWORD=39fdfgk4d3fdovszc932456j2oZ

# Provide an instance uuid to rescue and reset windows password

USERNAME=mycloudusernamehere
APIKEY=myapikeyhere
# DDI is the 'customer ID', if you don't know this login to the control panel and check the number in the URL
DDI=10010101
# The instance uuid you want to rescue
INSTANCE=ca371a8b-748e-46da-9e6d-8c594691f71c

# INITIATE RESCUE PROCESS

nova  --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure rescue --password "$PASSWORD" --image 7fade26a-0cca-415f-a988-49c021768fca $INSTANCE

# LOOP UNTIL STATE DETECTED AS RESCUED

STATE=0
until [[ $STATE == rescued ]]; do
echo "start rescue check"
STATE=`nova --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure show $INSTANCE | grep rescued | awk '{print $4}'`

echo "STATE =" $STATE
echo "sleeping.."
sleep 5
done

# EXTRACT PUBLIC ipv4 FROM INSTANCE

IP=`nova --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure show $INSTANCE | grep public | awk '{print $5}' | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])'`
echo "IP = $IP"

# UPDATE AND INSTALL RESCUE TOOLS AND RESET WINDOWS PASS
# Set environment locally
yum install sshpass -y

# Execute environment remotely
echo "Performing Rescue..."
sshpass -p "$PASSWORD" ssh -o StrictHostKeyChecking=no root@"$IP" 'yum update -y; yum install ntfs-3g -y; mount /dev/xvdb1 /mnt; curl li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm -o /root/nux.rpm; rpm -Uvh /root/nux.rpm; yum install chntpw -y; cd /mnt/Windows/System32/config; echo -e "1\ny\n" | chntpw -u "Administrator" SAM'

echo "Unrescuing in 100 seconds..."
sleep 100
nova  --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure unrescue $INSTANCE

Thanks again to my friend Cory who gave me the instructions, I simply automated the process to make it easier and learned something in the process 😉

Disabling SELinux

Posted on 5th May 2016 by azio

Today we had a customer that needed to perform a first generation server to next generation migration however they cannot have SELinux enabled during this process.

I explain to the customer how to disable this, it’s pretty simple.

vi /etc/sysconfig/selinux

SELINUX=enforcing

Needs to be changed to

SELINUX=disabled

Job done. A simple one but nonetheless important stuff. If you wanted to automate this it would look something like this;

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g'/etc/selinux/config

This sed oneliner simply swaps SELINUX=enforcing to SELINUX=disabled, pretty simple stuffs. It will work on CentOS6 and 7 anyway, and should but I can’t guarantee work on CentOS 5.

Reset Windows Administrator Password on Rackspace Cloud Server using Rescue Mode (without nova-agent)

Posted on 29th April 2016 by azio

So, you have lost your Windows Administrator password for your Rackspace cloud server? I’d like to thank my friend Cory for providing the link details for how to do this.

No problem. Simply put the Windows VM into rescue mode using a Linux image (yup!)

Put Windows VM into Rescue mode using Linux image

# Initiate rescue using the CentOS 7 image for the server uuid 0b67faf7-bc56-4844-ad0b-16e39f289ef6
$ nova me rescue --password mypasswordforrescuemodehere --image 7fade26a-0cca-415f-a988-49c021768fca 0b67faf7-bc56-4844-ad0b-16e39f289ef6

If you’ve broken your Rackspace server and you don’t know how to perform the above step, send a ticket to Rackspace support and they should be able to put your server in rescue so you can reset the password of your windows machine!

SSH to rescue server

ssh root@myserveriphere

Check which disk is Windows NTFS

# fdisk -l

Disk /dev/xvdc: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0003e9b3

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1            2048     4194303     2096128   83  Linux

Disk /dev/xvdb: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xfcb073fc

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1   *        2048   167770111    83884032    7  HPFS/NTFS/exFAT

Disk /dev/xvda: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00070dc0

Here we can see that the disk we want is /dev/xvdb1 since this is the HPFS/NTFS/exFAT partition format used by windows. The rescue mode builds a new server and disk, attaching your old disk as the ‘b’ disk, xvdb. Lets mount the disk and install the application we need to wipe the password for the box.

Mount the disk

yum update -y
yum install ntfs-3g -y
mount /dev/xvdb1 /mnt

Download and install the chntpw tool

 
curl li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm -o /root/nux.rpm
rpm -Uvh /root/nux.rpm
yum install chntpw -y

Run tool against Windows Administrator SAM file

Once run press 1 to ‘clear password’.

root@RESCUE-test config]# chntpw -u "Administrator" SAM
chntpw version 0.99.6 110511 , (c) Petter N Hagen
Hive  name (from header): <\SystemRoot\System32\Config\SAM>
ROOT KEY at offset: 0x001020 * Subkey indexing type is: 666c 
File size 262144 [40000] bytes, containing 6 pages (+ 1 headerpage)
Used for data: 255/20712 blocks/bytes, unused: 13/3672 blocks/bytes.


* SAM policy limits:
Failed logins before lockout is: 0
Minimum password length        : 0
Password history count         : 0
| RID -|---------- Username ------------| Admin? |- Lock? --|
| 01f4 | Administrator                  | ADMIN  |          |
| 01f5 | Guest                          |        | dis/lock |

---------------------> SYSKEY CHECK <-----------------------
SYSTEM   SecureBoot            : -1 -> Not Set (not installed, good!)
SAM      Account\F             : 0 -> off
SECURITY PolSecretEncryptionKey: -1 -> Not Set (OK if this is NT4)
Syskey not installed!

RID     : 0500 [01f4]
Username: Administrator
fullname:
comment : Built-in account for administering the computer/domain
homedir :

User is member of 1 groups:
00000220 = Administrators (which has 1 members)

Account bits: 0x0010 =
[ ] Disabled        | [ ] Homedir req.    | [ ] Passwd not req. |
[ ] Temp. duplicate | [X] Normal account  | [ ] NMS account     |
[ ] Domain trust ac | [ ] Wks trust act.  | [ ] Srv trust act   |
[ ] Pwd don't expir | [ ] Auto lockout    | [ ] (unknown 0x08)  |
[ ] (unknown 0x10)  | [ ] (unknown 0x20)  | [ ] (unknown 0x40)  |

Failed login count: 0, while max tries is: 0
Total  login count: 15

- - - - User Edit Menu:
 1 - Clear (blank) user password
 2 - Edit (set new) user password (careful with this on XP or Vista)
 3 - Promote user (make user an administrator)
(4 - Unlock and enable user account) [seems unlocked already]
 q - Quit editing user, back to user select
Select: [q] > 1
Password cleared!

Hives that have changed:
 #  Name
 0  
Write hive files? (y/n) [n] : y
 0   - OK

It’s been done, yay!

Unrescue the cloud server, either from control panel or using nova

abull-mb:~ adam$ supernova me unrescue 0b67faf7-bc56-4844-ad0b-16e39f289ef6

Yay! We now automatically bypass the ordinary login screen so we can get into the server to reconfigure it properly again.

You might have some questions about… setting up nova.

Setting up Nova

# Nova configuration

#export OS_AUTH_URL=https://lon.identity.api.rackspacecloud.com/v2.0/
#export OS_AUTH_SYSTEM=rackspace_uk
#export OS_REGION_NAME=LON
#export OS_USERNAME=mycloudusernamehere
# Tenant Name is customer number shown in url of mycloud control panel
##export OS_TENANT_NAME=10101010
#export NOVA_RAX_AUTH=1
#export OS_PASSWORD=mycloudapikeyhere
# Project ID is customer number shown in url of mycloud control panel
#export OS_PROJECT_ID=100101010
#export OS_NO_CACHE=1

These ‘environment variables’ should be put in a file like your .bash_profile. Then you will want to source it before using nova

source .bash_profile
or
. .bash_profile

This just sets the variables on the commandline so they can be used by nova. It is possible to provide all of the credentials on the nova commandline as described in previous articles on this blog concerning nova.

Using nova without .bash_profile or environment variables

Initiate Rescue Mode

nova --os-username mycloudusernamegoeshere --os-auth-system=rackspace  --os-tenant-name tenantidgoeshere --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password apigoeshere rescue --password mypasswordforrescuemodehere --image 7fade26a-0cca-415f-a988-49c021768fca 0b67faf7-bc56-4844-ad0b-16e39f289ef6

Un-rescue

nova --os-username mycloudusernamegoeshere --os-auth-system=rackspace  --os-tenant-name tenantidgoeshere --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password apigoeshere unrescue

Installing nova

for more details about how to install python based nova, used in this article, please see;
https://support.rackspace.com/how-to/installing-python-novaclient-on-linux-and-mac-os/

Installing KVM, libvirtd virt-manager and Xenserver for Rackspace onmetal using ZFS & X11 Forwarding

Posted on 27th April 2016 by azio

So, you want to run your own hypervisor using xenserver, but you want to have some of the flexibility of KVM too. This instructional guide explains how to install and configure KVM with virt-manager and with X11 forwarding. We will go step by step. In this case I am using a mac.

Step 1 – Create Rackspace onmetalv2 server

In this case I’ll be using a 40 cpu 128GB machine as the host utilizing the new onmetalv2 server range offered by Rackspace public cloud.

Please note that this is a bare metal server, not a cloud server, however it is offered by the same cloud platform at mycloud.rackspace.co.uk

Step 2 – Install and configure KVM

sudo yum update -y
sudo yum -y install kvm virt-manager libvirt virt-install qemu-kvm xauth dejavu-lgc-sans-fonts

Step 3 – Start and configure libvirtd

chkconfig libvirtd
service libvirtd status
service libvirtd restart
service libvirtd status

Step 4 – MAC SYSTEMS – Install X Quartz

For mac users simply install X Quartz, which can be found at http://www.xquartz.org/

Step 4a – Windows Systems – Install Xming

Windows users can get in on the action too, using xming which can be found at https://sourceforge.net/projects/xming/

Step 5 – MAC SYSTEMS ONLY – Configure X11 Forwarding

Xming will work out of the box for windows, but for Mac users you need to make sure you have enabled X11 forwarding.

touch ~/.ssh/config
echo "ForwardX11 yes" >> ~/.ssh/config

This simply allows X11 forwarding for Mac users which needs to be done at the client side. Then you can virtualize any application you like on the client, but running the application such as firefox , or even a virtual machine on the remote server. SSHv2 is beautiful. That’s it you’ve completed the most important steps.

Running virt-manager for the first time

 
[root@on-metal-test-2 ~]# virt-manager

After running the above command you will see something like the image below. You’ll see an X window open on your local client machine, which is associated with an application running on the remote server your connected to via SSH. This is pretty damn cool.

Lets take this further and install firefox to demonstrate how awesome this is!

yum install firefox -y

Now we’re using firefox thru ssh, much better and more convenient to use X11 forwarding for this, than using a proxy for instance on the client configured with tunnel or vpn.

Nice!

Lets take it a bit further and start installing xen server with KVM. I am very tempted to use ZFS for this since onmetal v2 has 2 1600GB disks…

Create partitions for KVM store

fdisk -l 
fdisk /dev/sdc

# type m , then type n, then type p, enter, enter, enter, enter, then type w

fdisk /dev/sdd

# type m , then type n, then type p, enter, enter, enter, enter, then type w

Create filesystem for KVM store

[root@on-metal-test-2 ~]# mkfs.ext3 /dev/sdc1 && mkfs.ext3 /dev/sdd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
97656832 inodes, 390624640 blocks
19531232 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
11921 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
97656832 inodes, 390624640 blocks
19531232 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
11921 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now we have created the filesystem. What about creating the ZFS partition. To do this we need to go thru a fairly laborious process (at least if you don’t know what your doing). As I discovered my yum installation wasn’t automatically providing the correct devel source for the kernel to use the ZFS DKMS module. As ZFS is really a native BSD package.

One of the problems I had was this

Loading new spl-0.6.5.6 DKMS files...
Building for 3.10.0-327.10.1.el7.x86_64
Module build for kernel 3.10.0-327.10.1.el7.x86_64 was skipped since the
kernel source for this kernel does not seem to be installed.
  Installing : zfs-dkms-0.6.5.6-1.el7.centos.noarch                                                                                                                       4/6
Loading new zfs-0.6.5.6 DKMS files...
Building for 3.10.0-327.10.1.el7.x86_64
Module build for kernel 3.10.0-327.10.1.el7.x86_64 was skipped since the
kernel source for this kernel does not seem to be installed.

This can be checked out in more detail by running an;

yum search --show-duplicates kernel-devel
# and
rpm -qa | grep kernel

This gave me the right version of the devel kernel I needed to install ZFS to my current kernel with a module, as opposed to completely recompiling the whole thing. Nice!

Install ZFS and kernel devel

sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm
sudo yum install epel-release

sudo yum install zfs kernel-devel-3.10.0-327.10.1.el7.x86_64

Enable ZFS

[root@on-metal-test-2 adam]# /sbin/modprobe zfs

Create the 2 disk mirror using ZFS

[root@on-metal-test-2 adam]# zpool create -f kvmstore mirror sdc1 sdd1

Check KVM store disk

[root@on-metal-test-2 adam]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md126p1    220G  2.3G  209G   2% /
devtmpfs         63G     0   63G   0% /dev
tmpfs            63G     0   63G   0% /dev/shm
tmpfs            63G   26M   63G   1% /run
tmpfs            63G     0   63G   0% /sys/fs/cgroup
tmpfs            13G  4.0K   13G   1% /run/user/0
kvmstore        1.5T     0  1.5T   0% /kvmstore

Run Virt manager to create Xenserver VM

Now we’ve created our partition and filesystem and configured ZFS we can run the virtual machines off the new kvm partition store. simples

Click top left icon on corner to create new VM

Download the Xenserver ISO to /root of hypervisor

root@on-metal-test-2 ~]# wget http://downloadns.citrix.com.edgesuite.net/10175/XenServer-6.5.0-xenserver.org-install-cd.iso
--2016-04-27 10:29:22--  http://downloadns.citrix.com.edgesuite.net/10175/XenServer-6.5.0-xenserver.org-install-cd.iso
Resolving downloadns.citrix.com.edgesuite.net (downloadns.citrix.com.edgesuite.net)... 104.86.110.32, 104.86.110.49
Connecting to downloadns.citrix.com.edgesuite.net (downloadns.citrix.com.edgesuite.net)|104.86.110.32|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 603744256 (576M) [application/octet-stream]
Saving to: ‘XenServer-6.5.0-xenserver.org-install-cd.iso’

100%[====================================================================================================================================>] 603,744,256 17.6MB/s   in 38s

Select Local Media (we’re going to use a Xenserver ISO)

Click browse, then press the bottom left + icon to add some pools. We’re going to add /root which has our iso in it, and we’re also going to add kvmstore aswell.

Congratulations you have now added the stores. Now all we need to do is finish configuring the VM.

We want to select the root partition now we have set up the pool, and choose the xenserver iso we just recently downloaded.

We are almost there now! Lets set the number of cpu and ram! Also lets make sure we use the kvmstore we just setup instead of the ‘main disk’ of the server.

Select our KVM store ‘pool’ on the left hand side, and then press + to add the kvmstore.qcow2 volume, see the images for illustration.

click choose volume at the bottom left to confirm! And finally name the server

awwww crap , we got this error because the libvirtd kvm configuration isnt running as root

This can be quickly resolved by editing the /etc/libvirt/qemu.conf and making sure user = “root” and group = “root” are present.

Job done!

Updating Citrix Xenserver 6.5 to SP1

Posted on 27th April 2016 by azio

I recently installed Xenserver on Rackspace OnMetal v2 and I wanted to make sure I had the latest supporting service packages.

It’s simple enough, just wget the update from citrix on their page and then patch-upload and patch-apply using the uuid of the patch and the uuid of the host. Simples.

Download Support Pack from Citrix

# wget -c http://downloadns.citrix.com.edgesuite.net/10340/XS65ESP1.zip

Unzip Support Pack

[root@xenserver-skucvxib ~]# unzip XS65ESP1.zip
Archive:  XS65ESP1.zip
  inflating: XS65ESP1.xsupdate
  inflating: XS65ESP1-src-pkgs.tar.bz2

Delete unused files

[root@xenserver-skucvxib ~]# rm XS65ESP1.zip
rm: remove regular file `XS65ESP1.zip'? y

Upload patch to Xen (and obtain the patch uuid)

[root@xenserver-skucvxib ~]# xe patch-upload file-name=XS65ESP1.xsupdate
7f2e4a3a-4098-4a71-84ff-b0ba919723c7

Remove all unnecessary files

[root@xenserver-skucvxib ~]# rm XS65ESP1*
rm: remove regular file `XS65ESP1-src-pkgs.tar.bz2'? y
rm: remove regular file `XS65ESP1.xsupdate'? y

Obtain uuid of host

root@xenserver-skucvxib ~]# xe host-list
uuid ( RO)                : 533a880b-6dd9-4d65-b930-55a3e4b27668
          name-label ( RW): xenserver-skucvxib
    name-description ( RW): Default install of XenServer

Apply patch

[root@xenserver-skucvxib ~]# xe patch-apply uuid=7f2e4a3a-4098-4a71-84ff-b0ba919723c7 host-uuid=533a880b-6dd9-4d65-b930-55a3e4b27668
5ee785d4-fc7d-dfe4-b250-26346b88898b is the local tools SR: scanning
Done
Preparing...                ##################################################
xen-device-model            ##################################################
Preparing...                ##################################################
xen-hypervisor              ##################################################
Preparing...                ##################################################
xen-tools                   ##################################################
Preparing...                ##################################################
xen-libs                    ##################################################
Preparing...                ##################################################
xen-dom0-tools              ##################################################
Preparing...                ##################################################
xen-dom0-libs               ##################################################
Preparing...                ##################################################
xen-ocaml-libs              ##################################################
Preparing...                ##################################################
guest-templates             ##################################################
Preparing...                ##################################################
lvm2                        ##################################################
Preparing...                ##################################################
upgrade-plugin              ##################################################
Preparing...                ##################################################
xapi-core                   ##################################################
Preparing...                ##################################################
xapi-xenopsd                ##################################################
Preparing...                ##################################################
xapi-rrdd                   ##################################################
Preparing...                ##################################################
Stopping XCP RRDD plugin xcp-rrdd-iostat: [  OK  ]
Stopping XCP RRDD plugin xcp-rrdd-squeezed: [  OK  ]
Stopping XCP RRDD plugin xcp-rrdd-xenpm: [  OK  ]
rrdd-plugins                ##################################################
Starting XCP RRDD plugin xcp-rrdd-iostat: [  OK  ]
Starting XCP RRDD plugin xcp-rrdd-squeezed: [  OK  ]
Starting XCP RRDD plugin xcp-rrdd-xenpm: [  OK  ]
Preparing...                ##################################################
blktap                      ##################################################
Preparing...                ##################################################
sm                          ##################################################
Preparing...                ##################################################
tzdata                      ##################################################
Preparing...                ##################################################
kernel                      ##################################################
unable to stat /sys/class/block//var/swap/swap.001: No such file or directory
Preparing...                ##################################################
vgpu                        ##################################################
Preparing...                ##################################################
linux-guest-loader-data     ##################################################
Preparing...                ##################################################
hwdata                      ##################################################
Preparing...                ##################################################
xenserver-transfer-vm       ##################################################
Preparing...                ##################################################
openvswitch                 ##################################################
Preparing...                ##################################################
v6d                         ##################################################
Preparing...                ##################################################
glibc                       ##################################################
Preparing...                ##################################################
glibc-common                ##################################################
Waiting for xapi to signal init complete
Removing any existing built-in templates
Regenerating built-in templates

We have now successfully applied Xen server SP1 to Xen Server 6.5.

Virt-manager won’t release mouse on Mac OS X

Posted on 26th April 2016 by azio

This was quite annoying, but thanks to Major hayden (thanks pal) I was able to resolve this issue by making a file on my Mac to make sure the bindings are there.

Make sure you close X Quartz first.

touch ~/.Xmodmap 
echo "clear Mod1" >> ~/.Xmodmap 
echo "keycode 66 = Alt_L"  >> ~/.Xmodmap 
echo "keycode 69 = Alt_R" >> ~/.Xmodmap 
echo "add Mod1 = Alt_L" >> ~/.Xmodmap 
echo "add Mod1 = Alt_R" >> ~/.Xmodmap

Job done! Now it works nicely 😀 This might be relevant to those using xen, kvm and libvirtd in particular.

Protecting yourself from Brute Force Attacks on SSH

Posted on 26th April 2016 by azio

To protect yourself against this type of attack succeeding, namely to guess your password and compromise your server, you can choose an extremely long password utilizing many symbols %$!&^ and numbers 190921, both UPPERCASE and lowercase letters.

Alternatively, and the most effective way to protect yourself against these attacks is to change the port with which your SSH service runs on. However some caution should be taken when changing SSH ports of the server, as if done improperly this can cause you to lock yourself out with the firewall. The process is simple though;

locate sshd_config in /etc/ssh/sshd_config

vi /etc/ssh/sshd_config

Change port to something else like 777

Change from:

#Port 22

Change to:

Port 777

Open up Firewall rule (very important)

sudo iptables -I INPUT 1 -p tcp  --dport 777 -j ACCEPT

Save Firewall Rule

/etc/init.d/iptables save

The entire process is described at:
https://support.rackspace.com/how-to/introduction-to-iptables/

Other alternatives ways to resolve this other than using secure passwords, or a less obvious port than SSH’s default port 22, is to install something like fail2ban. Which will ban any IP address preventing it from logging in if it gets the password wrong a certain number of times in a row. However, pleas be careful with this too as you can lock yourself out if you are not careful, as with all security software.

Backing up a MySQL Database remotely

Posted on 25th April 2016 by azio

So, you might want to backup a MySQL database remotely, like one of our customers did today. This is relatively simply utilizing the inbuilt mysqldump facility. This customer in particular was running varnish in front of his apache2 webserver so setting up phpmyadmin wasn’t entirely straight forward for this non technical customer. It’s easily achievable with something like;

Specific database

ssh -l user 1.1.1.1 "mysqldump -mysqldumpoptions databasenamegoeshere | gzip -3 -c" > /localpath/localfile.sql.gz

All databases

mysqldump -uroot -ppassword -h162.13.137.249 > backup.sql

The formatting of the command should look like

mysqldump -u root -p[root_password] -h [hostname] [database_name] > dumpfilename.sql