Checking a Website or Rackspace Load Balancers Supported SSL Ciphers

Posted on 11th October 2016 by azio

So, you may have recently had an audit performed, or have been warned about the dangers of SSLv3, poodle attack, heartbleed and etc. You want to understand exactly which ciphers your using on the Load Balancer, cloud-server, or dedicated server. It’s actually very easy to do this with nmap. Install it first, naturally.

# CentOS / RedHat
yum install nmap

# Debian / Ubuntu
apt-get install nmap

# Check for SSL ciphers

# nmap hostnamegoeshere.com --script ssl-enum-ciphers -p 443

Nmap done: 1 IP address (1 host up) scanned in 1.57 seconds

In this case we can see that only TLS v1.1 and TLS v1.0 are supported. No TLSv1.2 and no SSLv3.

Cheers &
Best wishes,
Adam

Troubleshooting Rackspace CDN not serving files

Posted on 11th October 2016 by azio

A customer came to us with an issue with their CDN which was strange and odd. I wanted to document this so that it is understood why this happened.

The customer is using two TLS origins, and HTTP2. Why that is a problem will become evidently clear. This is a general method of troubleshooting in terms of replicating behaviour of the CDN and origin with host headers. This can be applied no matter what the problem, to understand the HTTP code given by the origin, which at least half of the time turns out to be the cause. The origin being the cloud-server your CDN is backed by.

Question
Hi,
We are currently experiencing some issues with the Cloud CDN. We are using this for our CSS and images and now everything is getting a HTTP/503 SERVICE UNAVAILABLE. If you want to test, you may test this url:
https://cdn.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css

This is supposed to deliver this file:
https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css

Is something mis-configured or are there some issues on the appliance?

Answer

First we confirm the origin is UP

# curl -I https://originserver.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css
HTTP/1.1 200 OK
Date: Tue, 11 Oct 2016 08:45:42 GMT
Server: Apache
Last-Modified: Tue, 11 Oct 2016 06:57:53 GMT
ETag: "ed26-53e91653c61d0"
Accept-Ranges: bytes
Content-Length: 60710
Vary: Accept-Encoding
Cache-Control: max-age=31536000, public
Expires: Wed, 11 Oct 2017 08:45:42 GMT
Access-Control-Allow-Origin: *
X-Frame-Options: SAMEORIGIN
Content-Type: text/css

The origin is the cloud-server where the CDN pulls from. As we can see the site is up. So what is causing the issue? The way CDN works is it provides a host header for the domain, so the site has to have a host for both domains. The reason is that the CDN uses CNAME hostnames to identify which CDN is which. I.e. which path like /media/ directs to which static origins subdomain.

The best way to look further at the situation now is to check the origin (the subdomain that you’ve associated with your CDN subdomain that raxspace gives you, when sending the host header for the CDN url we get:

root@myweb:~# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: cdn.cusomerdomain.no'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:17:38 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

As we can see we get this odd HTTP 421 misdirected request.

# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: mycdnname1.scdn4.secure.raxcdn.com'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:18:06 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

~# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: cdncustomercname.cusomerdomain.com'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:17:45 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

https://httpd.apache.org/docs/2.4/mod/mod_http2.html

Looking at the definition for HTTP 2, this issue was caused by different TLS configurations for your domains and mod http2 trying to reuse the same connection, which will not work if the TLS configurations are not the same on the origin cloud-server side.

You just need to disable HTTP2, or configure the TLS configurations to be the same on the apache2 side. I hope that this clarifies and makes sense to you, of course if you have additional questions, comments or concerns please don't hesitate to reach out to us, we are here to help!

As you can see the importance of debugging CDN by sending host header to the origin that the CDN uses, to replicate the issue the customer was experiencing, which was essentially, the CDN edgenodes (the machines around the world that pull from the origin for content distribution really worldwide), weren't able to retrieve the files from the origin with the host header domain that is defined in the control panel.

This customer needed to in this case adjust their Apache2 configuration. The problem was likely caused by updating Apache2 or similar.

Rackspace Cloud Server not coming up after building it

Posted on 6th October 2016 by azio

So! Perhaps you’ve read my article on nova-agent , the common cause of this issue? If you haven’t you should since it covers out well the importance of nova-agent.

However, nova-agent itself also comes unstuck if the machine nova-agent is installed on is missing the xe-linux-distribution service, this service is provided by the package xe-guest-utilities and can be installed by yourselves, in the case that installing/ ensuring nova-agent starts on boot does not fix your issues.

Specifically if your nova-agent log provides you this message, you know you need to install the xe-guest-utilities. Simplies!

Problem

# cat /var/log/nova-agent.log

2016-10-06 18:58:14,696 [ERROR] [EXC] Traceback (most recent call last):
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/nova-agent.py", line 40, in 
2016-10-06 18:58:14,697 [ERROR] [EXC]     xs = plugins.XSComm()
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/1.39.0/plugins/xscomm.py", line 43, in __init__
2016-10-06 18:58:14,697 [ERROR] [EXC]     self.xs_handle = pyxenstore.Handle()
2016-10-06 18:58:14,700 [ERROR] [EXC] PyXenStoreError: Couldn't open connection to the xenstore: No such file or directory
2016-10-06 18:58:14,701 [ERROR] failed to parse config file '/usr/share/nova-agent/nova-agent.py'

Solution

# Redhat and CentOS systems
yum install xe-guest-utilities

# Debian, Ubuntu and other apt based systems
apt-get install xe-guest-utilities

I hope that this is of some assistance, here is some more background information.

More details about nova-agent and xe-guest-utilities in Xen

Provided that you have definitely enabled nova-agent, and ensured that it is running (after restarting the original server), with ps auxfwww | grep nova-agent

then, you should be good to re-image the original server, and then rebuild out the second.

The reason why your server doesn’t appear to be coming up in the new build is for some reason, nova-agent service got disabled on boot-time, and as a result, the nova-agent service responsible for swapping out the network configuration of your cloud-server wasn’t started up when the server was built, and the automatic ip configuration change didn’t occur. This explains well the behaviour you’ve been seeing, and after looking in the backend the error code seems to confirm that the issue was that the nova-agent wasn’t running.

Provided that you’ve definitely installed nova-agent and confirmed it is running, as well as made sure it starts at boot time, as in the article I wrote, you should be good.

I hope that this explanation and clarification meets you well.

I can see that you’ve recently posted an additional issue that has been experienced with xe-linux-distribution (the cause of the PyXenStoreError). This secondary cause of the issue can be fixed by ensuring xe-linux-distribution is installed;

apt-get update;
apt-get install xe-guest-utilities

This should install the xen guest tools as required by nova-agent. This is required by the nova-agent in order for the networking data to be retrieved by nova-agent, whereas nova-agent itself applies the change, but these services both need to be running and installed for this to work properly!

I really hope that this is of some assistance ,of course if you have additional questions, comments or concerns please don’t hesitate to write back, and we can escalate this issue further for you. These instructions should fix your issues though! I hope this helps &

Adding AuthType Basic HTTP Authentication for WordPress wp-admin , phpMyAdmin, etc

Posted on 3rd October 2016 by azio

Hey folks, so this comes up really often with Rackspace customers, how to add AuthType Basic HTTP authentication. This is akin to an additional user/pass dialog box which appears when navigating to the url. It is really easy to create a new password:

htpasswd -c /etc/httpd/.htpasswd-private username

Please note .htpasswd-private can be anything, however it’s better to make it a hidden file by prefixing with a . dot, also important to ensure that the password file isn’t in a web documentroot, the file should have user and group root root, but that should be done automatically when you create the file with root user.

This is what I added to phpmyadmin configuration file in /etc/httpd/conf.d/phpMyAdmin.conf

 
<Location "/phpMyAdmin/">
AuthType Basic
AuthName "Private Server"
AuthUserFile /etc/httpd/.htpasswd-private
Require valid-user
</Location>

You could do the same for WordPress;

<Location "/wp-admin/">
AuthType Basic
AuthName "Private Server"
AuthUserFile /etc/httpd/.htpasswd-private
Require valid-user
</Location>

Please note that I am using Location, but you can apply the same thing within a directive instead.

Could not resolve mirror.rackspace.com

Posted on 20th September 2016 by azio

So a customer came to me with an issue about not being able to install chkrootkit, due to DNS failure. The customer wasn’t technical and didn’t understand how to fix it. It’s easy to fix, by wiping the resolv.conf nameservers file, and rebuilding it. Takes a few seconds, this should allow you to resolve domains afterwards again.


# Delete resolv.conf

rm /etc/resolv.conf

# Recreate the DNS file with different DNS servers
touch /etc/resolv.conf
echo "nameserver 4.2.2.4" >> /etc/resolv.conf
echo "nameserver 8.8.8.8" >> /etc/resolv.conf

Alternatively you could just open it up for editing with vi or nano.

vi /etc/resolv.conf
nano /etc/resolv.conf

Securing your WordPress with chmod 644 and chmod 755 the easy (but pro) way

Posted on 16th September 2016 by azio

Let’s say we have a document root like:

It’s interesting to note the instructions for this will vary from environment to environment, it depends on which user is looking after apache2, etc.

/var/www/mysite.com/htdocs

Make all files read/write and owned by www-data apache2 user only

root@meine:/var/www/mysite.com/htdocs# find . -type f -exec chown apache2:apache2 {} \; 
root@meine:/var/www/mysite.com/htdocs# find . -type f -exec chmod 644 {} \;

Make all folders accessible Read + Execute, but no write permissions

root@meine:/var/www/mysite.com/htdocs# find . -type d -exec chmod 755 {} \;
root@meine:/var/www/mysite.com/htdocs# find . -type d -exec chown apache2:apache2 {} \;

PLEASE NOTE THIS BREAKS YOUR WORDPRESS ABILITY TO AUTO-UPDATE ITSELF. BUT IT IS MORE SECURE 😀

Note debian users, may need to use www-data:www-data instead.

Fixing Innodb Registered as a storage Engine Failed

Posted on 14th September 2016 by azio

So you have a nice wordpress (or similar site) running, but then all of a sudden you got a nasty message telling you that, the InnobDB registration as a storage engine failed.

# tail /var/log/mariadb/mariadb.log
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
160914  6:45:29 [ERROR] Plugin 'InnoDB' init function returned error.
160914  6:45:29 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160914  6:45:29 [ERROR] Failed to initialize plugins.
160914  6:45:29 [ERROR] Aborting

160914  6:45:29 [Note] /usr/libexec/mysqld: Shutdown complete

160914 06:45:29 mysqld_safe mysqld from pid file /var/lib/mysql/wpstack.localdomain.pid ended

# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Wed 2016-09-14 06:25:38 UTC; 2min 32s ago
  Process: 1434 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 1433 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 1305 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 1433 (code=exited, status=0/SUCCESS)

Sep 14 06:25:35 wpstack.localdomain systemd[1]: Starting MariaDB database se....
Sep 14 06:25:37 wpstack.localdomain mysqld_safe[1433]: 160914 06:25:37 mysqld...
Sep 14 06:25:37 wpstack.localdomain mysqld_safe[1433]: 160914 06:25:37 mysqld...
Sep 14 06:25:38 wpstack.localdomain systemd[1]: mariadb.service: control pro...1
Sep 14 06:25:38 wpstack.localdomain systemd[1]: Failed to start MariaDB data....
Sep 14 06:25:38 wpstack.localdomain systemd[1]: Unit mariadb.service entered....
Sep 14 06:25:38 wpstack.localdomain systemd[1]: mariadb.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@wpstack ~]# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@wpstack ~]# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

This is naturally a problem since until it is resolved you have no database service running. It is fairly easy to resolve in most cases, by removing the log from var/lib/mysql

mv /var/lib/mysql/ib_logfile0{,.bak}  	   	 	   
mv /var/lib/mysql/ib_logfile1{,.bak}  	   	 	   
mv /var/lib/mysql/ibdata1{,.bak}

These simply move the files away, which allows InnoDB to continue operating. Please note that this may not always fix your issue and in some situations might result in data loss, so it is advisable to take a backup of the database filesystem before proceeding.

Automating Backups in Public Cloud using Cloud Files

Posted on 14th September 2016 by azio

Hey folks, I know it’s been a little while since I put an article together. However I have been putting together a really article explaining how to write bespoke backup systems for the Rackspace Community. It’s a proof of concept/demonstration/tutorial as opposed to a production application. However people looking to create custom cloud backup scripts may benefit from the experience of reading thru it.

You can see the article at the below URL:

https://community.rackspace.com/products/f/25/t/7857

Testing Rackspace Cloud-server Service-net Connectivity and creating an alarm

Posted on 23rd August 2016 by azio

So, the last few weeks my colleagues and myself have been noticing that there has been a couple of issues with the cloud-servers servicenet interface. Unfortunately for some customers utilizing dbaas instances this means that their cloud-server will be unable to communicate, often, with their database backend.

The solution is a custom monitoring script that my colleague Marcin has kindly put together for another customer of his own.

The python script that goes on the server:

Create file:

vi /usr/lib/rackspace-monitoring-agent/plugins/servicenet.sh

Paste into file:

#!/bin/bash
#
ping="/usr/bin/ping -W 1 -c 1 -I eth1 -q"

if [ -z $1 ];then
   echo -e "status CRITICAL\nmetric ping_check uint32 1"
   exit 1
else
   $ping $1 &>/dev/null
   if [ "$?" -eq 0 ]; then
      echo -e "status OK\nmetric ping_check uint32 0"
      exit 0
   else
      echo -e "status CRITICAL\nmetric ping_check uint32 1"
      exit 1
   fi
fi

Create an alarm that utilizes the below metric

if (metric["ping_check"] == 1) {
    return new AlarmStatus(CRITICAL, 'what?');
}
if (metric["ping_check"] == 0) {
    return new AlarmStatus(OK, 'eee?');
}

Of course for this to work the primary requirement is a Rackspace Cloud-server and an installation of Rackspace Cloud Monitoring installed on the server already.

Thanks again Marcin, for this golden nugget.

Checking if your Cloud Server VM run out of memory after it has happened