A lot of customers ask the question of how to have a data volume that can be incrementally increased in size vertically over a period of time. Here is how to setup a server like that from start to finish.

Step 1. Create Rackspace Cloud server

Click create server at bottom left once you are happy with the distribution you want to use:

Step 2. Create Cloud Block Storage Volumes. In this case I’m going to create 3 x 75 Gig disks.

Now your done creating your server and the volumes you are going to use with it. We could have just added 1 Cloud block storage volume, and added the others later, but for this demo, we’re going to show you how to extend the initial partition with the space capacity of the other 2.

Step 3. Attach your Cloud Block Storage Volumes to the server:

Step 4. Login to your Cloud Server

$ ssh [email protected]
The authenticity of host '37.188.1.1 (37.188.1.1)' can't be established.
RSA key fingerprint is 51:e9:e6:c1:4b:f8:24:9f:2a:8a:36:ec:bf:47:23:d4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '37.188.1.1' (RSA) to the list of known hosts.
Last login: Thu Jan  1 00:00:10 1970

Step 5. Run fdisk -l (list) to see attached volumes to server

Disk /dev/xvdc: 536 MB, 536870912 bytes, 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0004ece3

Device Boot Start End Blocks Id System
/dev/xvdc1 2048 1048575 523264 83 Linux

Disk /dev/xvda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b1244

Device Boot Start End Blocks Id System
/dev/xvda1 * 2048 41943039 20970496 83 Linux

Disk /dev/xvdb: 80.5 GB, 80530636800 bytes, 157286400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/xvdd: 80.5 GB, 80530636800 bytes, 157286400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I actually discovered at this point that CentOS 7 only supports 3 virtual disks as standard. I’m having the issue because the Rackspace centOS 7 image is shipping with HVM which is causing the issues, if it was just PV type we would be okay. You should switch to a PV version of CentOS now if you want more than 3 virtual disks with your Rackspace Cloud Server.

Step 5: Running the same command on a CentOS 6 PV server allows me to add more disks thru the control panel

[root@lvm-extend-test ~]# fdisk -l

Disk /dev/xvdc: 536 MB, 536870912 bytes
70 heads, 4 sectors/track, 3744 cylinders
Units = cylinders of 280 * 512 = 143360 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000f037d

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1               8        3745      523264   83  Linux

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003e086

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1        2611    20970496   83  Linux

Disk /dev/xvdb: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/xvdd: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/xvde: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/xvdf: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/xvdg: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Many disks are now available, we can see them by running:

[root@lvm-extend-test ~]# ls /dev/xv*
/dev/xvda  /dev/xvda1  /dev/xvdb  /dev/xvdc  /dev/xvdc1  /dev/xvdd  /dev/xvde  /dev/xvdf  /dev/xvdg

Step 6: Run cfdisk and start partitioning each disk.

cfdisk /dev/xvdb

Create New Partitionof Primary Partition Type

Using maximum space available

Using type 8E LVM Filesystem TYPE

Write partition data:

Step 7: Repeat this for any additional block storage disks you may have. I have a total of 5 CBS volumes, so I need to repeat this another 4 times.

<pre>

cfdisk /dev/xvdc
cfdisk /dev/xvde
cfdisk /dev/xvdf
cfdisk /dev/xvdg

Step 8: Verify that the partitions exist. (check each one has a 1 on the end now)

[root@lvm-extend-test ~]# ls /dev/xvd*
/dev/xvda  /dev/xvda1  /dev/xvdb  /dev/xvdb1  /dev/xvdc  /dev/xvdc1  /dev/xvdd  /dev/xvdd1  /dev/xvde  /dev/xvde1  /dev/xvdf  /dev/xvdf1  /dev/xvdg  /dev/xvdg1

Step 9: Install LVM

 yum install lvm2 

Step 10: Create first physical volume

[root@lvm-extend-test ~]# pvcreate /dev/xvdb1
  Physical volume "/dev/xvdb1" successfully created

Step 11: Check Physical Volume

 [root@lvm-extend-test ~]# pvdisplay
  "/dev/xvdb1" is a new physical volume of "75.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/xvdb1
  VG Name
  PV Size               75.00 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               7Vv8Rf-hRIr-b7Cb-aaxY-baeg-zVKR-BblJij 

Step 11: Create a volume group on the first physical volume and give it a name DataGroup00

[root@lvm-extend-test ~]# vgcreate DataGroup00 /dev/xvdb1
  Volume group "DataGroup00" successfully created

[root@lvm-extend-test ~]# vgdisplay
  --- Volume group ---
  VG Name               DataGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               75.00 GiB
  PE Size               4.00 MiB
  Total PE              19199
  Alloc PE / Size       0 / 0
  Free  PE / Size       19199 / 75.00 GiB
  VG UUID               Gm00iH-2a15-HO8K-Pbnj-80oh-E2Et-LE1Y2A

Currently the disk is 75GB. We now want to expand/extend with LVM the size of the disk. Doing this is simple enough.

Step 12: Extend Volume size with LVM

[root@lvm-extend-test ~]# vgextend DataGroup00 /dev/xvdd1
  Physical volume "/dev/xvdd1" successfully created
  Volume group "DataGroup00" successfully extended
[root@lvm-extend-test ~]# vgdisplay
  --- Volume group ---
  VG Name               DataGroup00
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               149.99 GiB
  PE Size               4.00 MiB
  Total PE              38398
  Alloc PE / Size       0 / 0
  Free  PE / Size       38398 / 149.99 GiB
  VG UUID               Gm00iH-2a15-HO8K-Pbnj-80oh-E2Et-LE1Y2A

Now we can see we got double the space! Lets keep extending it.

Step 13: Extend Volume size again with LVM some more.

[root@lvm-extend-test ~]# vgextend DataGroup00 /dev/xvde1
  Physical volume "/dev/xvde1" successfully created
  Volume group "DataGroup00" successfully extended
[root@lvm-extend-test ~]# vgdisplay
  --- Volume group ---
  VG Name               DataGroup00
  System ID
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               224.99 GiB
  PE Size               4.00 MiB
  Total PE              57597
  Alloc PE / Size       0 / 0
  Free  PE / Size       57597 / 224.99 GiB
  VG UUID               Gm00iH-2a15-HO8K-Pbnj-80oh-E2Et-LE1Y2A

[root@lvm-extend-test ~]# vgextend DataGroup00 /dev/xvdf1
  Physical volume "/dev/xvdf1" successfully created
  Volume group "DataGroup00" successfully extended
[root@lvm-extend-test ~]# vgdisplay
  --- Volume group ---
  VG Name               DataGroup00
  System ID
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               299.98 GiB
  PE Size               4.00 MiB
  Total PE              76796
  Alloc PE / Size       0 / 0
  Free  PE / Size       76796 / 299.98 GiB
  VG UUID               Gm00iH-2a15-HO8K-Pbnj-80oh-E2Et-LE1Y2A

[root@lvm-extend-test ~]# vgextend DataGroup00 /dev/xvdg1
  Physical volume "/dev/xvdg1" successfully created
  Volume group "DataGroup00" successfully extended
[root@lvm-extend-test ~]# vgdisplay
  --- Volume group ---
  VG Name               DataGroup00
  System ID
  Format                lvm2
  Metadata Areas        5
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                5
  Act PV                5
  VG Size               374.98 GiB
  PE Size               4.00 MiB
  Total PE              95995
  Alloc PE / Size       0 / 0
  Free  PE / Size       95995 / 374.98 GiB
  VG UUID               Gm00iH-2a15-HO8K-Pbnj-80oh-E2Et-LE1Y2A

Now we are at 374.98GB Capacity. 5 x 75GB. No problems at all! Imagine if you were doing this with 1000GIG volumes. You could put yourself together a pretty tight CBS. The thing i’d be worried bout was data loss though. So you’d want a server identical to this, with rsync setup across the two for some level of redundancy. and you’d want it, preferably in a completely different datacentre, too.

Last thing now. Actually creating the ext4 filesystem on this volumegroup. We’ve partitioned so that the disk can be used. We’ve created volume and group so that disks can be assigned to the OS as a disk. Now we need to format it with the filesystem. So lets take some steps to do that:

Step 13: Create Logical Volume and Verify

[root@lvm-extend-test ~]# lvcreate -l +100%FREE DataGroup00 -n data
  Logical volume "data" created.

[root@lvm-extend-test ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/DataGroup00/data
  LV Name                data
  VG Name                DataGroup00
  LV UUID                JGTRSg-JdNm-aumq-wJFC-VHVb-Sdm9-VVfp5c
  LV Write Access        read/write
  LV Creation host, time lvm-extend-test, 2015-10-12 11:53:45 +0000
  LV Status              available
  # open                 0
  LV Size                374.98 GiB
  Current LE             95995
  Segments               5
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Step 14: Make Volume Active

[root@lvm-extend-test ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "DataGroup00" using metadata type lvm2

Step 15: Create Filesystem on physical volume

[root@lvm-extend-test ~]# mkfs.ext4 /dev/mapper/DataGroup00-data
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
24576000 inodes, 98298880 blocks
4914944 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Step 16: Making a moint point folder

 mkdir /lvm-data

Step 17: Update your fstab (TAKE CARE) so that the disk is attached to the required on boot

[root@lvm-extend-test ~]# vi /etc/fstab

# Required line 
/dev/mapper/DataGroup00-data    /lvm-data           ext4    defaults        0 0

Step 18: Mount the LVM

[root@lvm-extend-test ~]# mount /lvm-data
[root@lvm-extend-test ~]#

There ya go! You have your 375GB Volume! You can extend this at any point! Just simply make a new CBS volume and then repeat the process of mounting it and then extending it.

So, I was figuring, because of all the brute force attacks on my servers that I would bother to install fail2ban. Something even better than this would be to change the port your SSH runs on..

Step 1. Install Fail2ban
Ubuntu and Debian Systems

apt-get install fail2ban

Redhat, Fedora and CentOS based Systems

yum install fail2ban

Step 2. Copy the reference config file and edit it in Vi (or nano)

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
vi /etc/fail2ban/jail.local

Step 3. Configure bantime (default 600seconds), and the max_retry (3 attempts).
If someone tries to connect 3 times or more with the wrong password, they’ll be added to IPTABLES DROP rule for 600 seconds

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

maxretry = 3

By default fail2ban starts banning people on SSH immediately, but I found it was also possible to configure fail2ban to block ip addresses attempting to brute force hack my email accounts, here is how I did it.

[sasl]

enabled  = false
port     = smtp,ssmtp,submission,imap2,imap3,imaps,pop3,pop3s
filter   = postfix-sasl
# You might consider monitoring /var/log/mail.warn instead if you are
# running postfix since it would provide the same log lines at the
# "warn" level but overall at the smaller filesize.
logpath  = /var/log/mail.log

It’s possible to alter this configuration but for most people the logpath for SSH is auth.log

[ssh]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

Step 4: Restart the Fail2ban service

# most init.d based systems
/etc/init.d/fail2ban restart
# some systemD systems
service fail2ban restart

So, today I was taking a look at Rackspace’s Load Balancers. I wanted to put together a small tutorial how to spin up multiple cloud servers and add them to a normal HTTP load balancer. This is traditionally a use case for sites that good a lot of traffic, and/or require great redundancy both at the load balancer, and at the server level. i.e. multiple ip addresses and hardware that is failover redundant. If a server or load balancer goes down, then there is provisions to allow a new load balancer or cloud server to take over.

It’s a simple setup.

Step 1. Creating 2 or more Cloud Servers from the mycloud.rackspace.co.uk Control Panel.

Create two servers, using the above process.

Step 2. I am using SSH Keys, so I provide my public ssh id_dsa.pub for SSH, (for a guide on making SSH keys, see my tutorial on this site)

Create two servers with SSH KEY AUTHENTICATION (optional), using the above process.

Step 3. Install httpd and netcat service, in my case I am using CENTOS 7, which is a nice secure version of RHEL.
RedHat/CentOS Distributions:

yum install httpd nc

Debian/Ubuntu Distributions:

 
apt-get install httpd netcat

Step 4. Creating A Rackspace Load Balancer

Step 5. Add 2 or more server nodes to the Load Balancer

Be sure to tick the number of servers you want to add. Please note, it’s possible to add servers to the Load Balancer that aren’t part of the rackspace Network. To do that you can use the ‘add external node’ button. Please note though, that the requests between load balancer and the destination machine goes over the public network interface. Whereas requests from the load balancer to other rackspace servers will always go thru the service net by default. (that is the local 10.x.x.x IP addresses and networks).

Step 6. Configure Cloud Server Firewall Settings to Accept port 80 and (optionally) port 443

# Allow negotiating of connections on port 80 incoming (HTTP), and port 443 incoming (HTTPS)
sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT
# Allow negotiated connections replies to reach us
sudo iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

Step 7. Restart apache2 / httpd service, and check localhost on port 80

service httpd restart
curl localhost

Step 8. Configure the Load Balancer Algorithm (optional). I wanted to use a round-robin approach, each request increments the serverid. So with 3 servers. It takes 3 incremental requests for serverid 1 to serve http request. With 3 servers, i t takes 4 incremental requests for serverid 2 to serve a request twice!

There are various different settings to use.

Step 9. All Done! You’ve configured the load balancer, and attached 2 cloud servers to the load balancer. Requests can come into the load balancer on port 80, and it sends requests to either server1 or server2 on the same port, 80. It is however possible to send requests to the cloud servers on a different port, which is covered in another article on this website. You could for instance create a server using 10 SSL certificates on a single IP, using different ports. Which could work out a lot cheaper than leasing 10 IPV4 or wrestling with your service provider for additional IP’s for SSL usage

Step 10. Test requests to the load balancer

In this setup I have 2 cloud server IP’s: 5.79.24.207 and 5.79.24.205. They both listen on port 80.

In this setup I have 1 load balancer IP address: 134.213.160.178

I can now connect to the load balancer running on http://134.213.160.178 and it forwards the connections to 5.79.24.207 or 5.79.24.205. If I wanted I could have added another 100 cloud servers to the load balancer, and each time I load a HTTP request to the load balancer, a different cloud server will respond. This allows very large numbers of transactions to go to a website, and for multiple servers to respond individually to seperate customers requesting thru the load balancer. In a production environment after confirming these changes I would then.

Step 11: SET DNS to point to the load balancer IP instead of cloud server IP.

A RECORD ADDED AS BELOW:
mywebsite.com -> 134.213.160.178 

Step 12: Confirm Load Balancer is working

For my purposes I am using the default www website for CENTOS 7 HTTPD /var/www/html/index.html . I changed server1 index.html to say only ‘server1’ and I changed server2 index.html to say only ‘server2’. This way I can check the load balancer is giving a different server every second request. It was:

Step 13: Testing for Client IP information with Netcat

tailf /var/log/httpd/access.log
10.190.255.250 - - [05/Oct/2015:10:14:37 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
0.190.255.250 - - [05/Oct/2015:10:14:39 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
10.190.255.250 - - [05/Oct/2015:10:14:40 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
10.190.255.250 - - [05/Oct/2015:10:14:42 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"

Presently we can see that requests reaching our servers from cloud load balancer are seeing only the load balancer IP. We need each apache httpd server to know the client IP of each request.

service httpd stop

So, I run netcat on port 80 to see the requests from the load balancer.

[root@server1 html]# nc -l 80
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0
X-Forwarded-For: 94.236.7.190
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Cache-Control: max-age=0
X-Forwarded-Proto: http
Accept-Language: en-US,en;q=0.5
Host: 134.213.160.178
If-Modified-Since: Mon, 05 Oct 2015 10:11:57 GMT
X-Cluster-Client-Ip: 94.236.7.190
Via: 1.1 542204-LON4WWSG01.secops.rackspace.com 0A02CC2D
X-Forwarded-Port: 80
If-None-Match: "8-52158be551fbf"
Accept-Encoding: gzip, deflate

As we can see there is an X-Forwarded-For header from the load balancer which does reach the cloud server, but by default apache doesn’t know about it and doesn’t put the X-Forwarded-For variable in the Logs, only the src_ip, which is presently the load balancers IP and not our client. So we need to make a small modification to the apache2 default httpd configuration:

Step 14: Modify Apache HTTPD Log ‘combined’ configuration

cat /etc/httpd/conf/httpd.conf

Inside the above file we see the directives:

  

    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    CustomLog "logs/access_log" combined

The part we want to change is very small

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

We want to add the section above %{X-Forwarded-For}i. This will include the real Client_IP before the Load Balancer IP in the server logs, so it looks like so:

94.236.7.190 10.190.255.250 - - [05/Oct/2015:11:01:11 +0000] "GET / HTTP/1.1" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"

Today, one of my fellow colleagues received a call about a server that had run out of memory. They sent a soft reboot, and because of that the process task hung. This is because the hypervisor compute node sends a message to the nova agent running on the guest virtual machine! If the guest virtual machine has run out of memory, it’s not possible for nova to receive that command, or, if it does, then the soft (software) reboot can fail, because there is not enough memory to fork the process.

This could have been avoided by issuing a hard reboot straight away, but in this case we needed to cancel the task and send a hard reboot. Here is what I did:

List all pending tasks on xen-server

# xe task-list

uuid ( RO)                : a9f84f3d-0b96-8da2-a1d1-f5b774cd9173
          name-label ( RO): VM.clean_reboot
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.275

Cancel a pending task on xen-server

xe task-cancel uuid=a9f84f3d-0b96-8da2-a1d1-f5b774cd9173

This sets the active_state back to normal and gets rid of the ‘pending soft reboot’, but we need to restart the server too.

Using supernova API to stop and restart the server

supernova lon stop serveruuidhere
supernova lon start serveruuidhere

and…The customer is back up online and running, yay!