Diagnosing a sick website getting 500,000 to 1 million views a day

Posted on 27th March 2018 by azio

So today I had a customer that had some woes. I don’t even think they were aware they were getting 504’s but I had to come up with some novel ways to

A) show them where teh failure happened
B) Show them the failed pages that failed to load (i.e. get a 504 gateway timeout)
C) show them the number of requests and how they changed based on the day of the outage, and a ‘regular normal’ day.
D) show them specific type of pages which are failing to give better idea of where the failure was

In this case a lot of the failures were .html pages, so it could be a cache was being triggered too much, it could be that their application was really inefficient, or in many cases, were catalog search requests which no doubt would scratch the db pretty nastily if the database or the query wasn’t refactored or designed with scalability in mind.

With all that in mind I explained to the customer, even the most worrysome (or woesome) of applications and frameworks, and even the most grizzly of expensive MySQL queries can be combatted, simply by a more adaptable or advanced cache mechanism. Anyway, all of that out of the way, I said to them it’s important to understand the nature of the problem with the application, since in their case were getting a load average of over 600.

I don’t have their solution,. I have the solution to showing them the problem. Enter the sysad, blazing armour, etc etc. Well, thats the way it’s _supposed_ to happen !

cat /var/log/httpd/access_log | grep '26/Mar' | grep 'HTTP/1.1" 50' | wc -l
26081

cat /var/log/httpd/access_log | grep '27/Mar' | grep 'HTTP/1.1" 50' | wc -l
2

So we can see 504’s the day before wasn’t an issue, but how many requests did the site get for each day comparatively?

[root@anon-WEB1 httpd]# cat access_log | grep '26/Mar' | wc -l
437598

[root@anon-WEB1 httpd]# cat access_log | grep '25/Mar' | wc -l
339445

The box received 25% more traffic, but even based from the figures in the SAR, cpuload had gone up 1500% beyond what the 32 cores on their server could do. Crazy. It must be because requests are getting queued or rather ‘building up’, and there are so many requests reaching apache, hitting the request for mysql, that either mysql formed a bottleneck and might need more memory, or, at this scale, a larger or smaller (probably larger) sized packet for the request, this can speed up significantly how fast the memory bucket fills and empties, and request queue gets killed. Allowing it to build up is going to be a disaster, because it will mean not just slow queries get a 504 gateway timeout, but also normal requests to regular html pages too (or even cached pages), since at that stage the cpu is completely overwhelmed.

I wrote a script,

to find a majority of the 504’s for the 26 Mar you can use this piece:

cat access_log | grep '26/Mar' | grep 'HTTP/1.1" 50' | awk {'print $7'}

to generate a unique list for your developer/team of pages which failed you can run:

cat access_log | grep '26/Mar' | grep 'HTTP/1.1" 50' | awk {'print $7'} | sort | uniq

To be honest. In the simplicity of this post somewhere, is a stroke of inspiration (if not ingenuity). Also it’s kind of hacky and crap, but, it does work and it is effective for doing the job.

AND that is What counts.

Enabling MySQL Slow Query Logs

Posted on 30th May 2017 by azio

In the case your seeing very long pageload times, and have checked your application. It always pays to check the way in which the database performs when interacting with the application, especially if they are on either same or seperate server, as these significantly affect the way that your application will run.


mysql> SET GLOBAL slow_query_log = 'ON' ;
Query OK, 0 rows affected (0.01 sec)

mysql> SET GLOBAL slow_query_log_file = '/slow_query_logs/slow_query_logs.txt';
Query OK, 0 rows affected (0.00 sec)

mysql> SET GLOBAL long_query_time = 5;
Query OK, 0 rows affected (0.00 sec)

Checking Webserver Logs and generating hit counts

Posted on 16th January 2017 by azio

So, I’ve been meaning to do this for a while. I’m sure your all familiar with this crappy oneliner where you’ll check the hits using wc -l against a webserver log for a given hour and use for i in seq or similar to get the last 10 minutes data, or the last 59 minutes. But getting hours and days is a bit harder. You need some additional nesting, and, it isn’t difficult for you to do at all!

for i in 0{1..9}; do echo "16/Jan/2017:06:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/soemsite.com-access_log | wc -l ; done

I improved this one, quite significantly by adding an additional for j, for the hour, and adding an additional 0 in front of {1..9}, this properly is matching the Apache2 log format and allows me to increment through all the hours of the day. to present ;-D All that is missing is some error checking when the last date in the file is, im thinking a tail and grep for the timecode from the log should be sufficient.

Here is the proud oneliner I made for this!

for j in 0{1..9}; do for i in 0{1..9} {10..59}; do echo "16/Jan/2017:$j:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/website.com-access_log | wc -l ; done; done

Upgrading PHP 5.3.29 to PHP 7 on Centos 6.8 Using Rackspace IUS Repo

Posted on 13th January 2017 by azio

These instructions only apply in specific cases. Specifically CentOS machines, running in the Rackspace Cloud, IUS the Rackspace provided repo, provides several things not usually available within the CentOS repo, without you manually compiling more recent versions. One of them is the latest version of PHP7.0 and PHP7.1.

I wanted to quickly document the process, since it is a relatively simple process, and, can actually be done without any maintenance window, if you know what your doing, with very minimal, (if any) disruption to running sites. an apachectl graceful, actually, should be enough. Since apachectl gracefully restarts apache httpd, the downtime you’ll see will be super minimal. Expect nobody to notice you upgraded to PHP7 if you do this right.

If you do this incorrectly, you will break the PHP installation, and worse, break all of the sites using mod_php. Lets take a look at the steps:

Step 1. Check available PHP modules provided by presently configured REPO

root@server3 ~]# yum search php7
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
============================================================================================================================= N/S Matched: php7 ==============================================================================================================================
php70u-debuginfo.x86_64 : Debug information for package php70u
php70u-ioncube-loader-debuginfo.x86_64 : Debug information for package php70u-ioncube-loader
php70u-pecl-amqp-debuginfo.x86_64 : Debug information for package php70u-pecl-amqp
php70u-pecl-apcu-debuginfo.x86_64 : Debug information for package php70u-pecl-apcu
php70u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php70u-pecl-igbinary
php70u-pecl-imagick-debuginfo.x86_64 : Debug information for package php70u-pecl-imagick
php70u-pecl-redis-debuginfo.x86_64 : Debug information for package php70u-pecl-redis
php70u-pecl-smbclient-debuginfo.x86_64 : Debug information for package php70u-pecl-smbclient
php70u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php70u-pecl-xdebug
php71u-debuginfo.x86_64 : Debug information for package php71u
php71u-pecl-apcu-debuginfo.x86_64 : Debug information for package php71u-pecl-apcu
php71u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php71u-pecl-igbinary
php71u-pecl-redis-debuginfo.x86_64 : Debug information for package php71u-pecl-redis
php71u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php71u-pecl-xdebug
sclo-php70-php-pecl-propro-devel.x86_64 : sclo-php70-php-pecl-propro developer files (header)
sclo-php70-php-pecl-raphf-devel.x86_64 : sclo-php70-php-pecl-raphf developer files (header)
uwsgi-plugin-php70u-debuginfo.x86_64 : Debug information for package uwsgi-plugin-php70u
mod_php70u.x86_64 : PHP module for the Apache HTTP Server
mod_php71u.x86_64 : PHP module for the Apache HTTP Server
php70u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php70u-cli.x86_64 : Command-line interface for PHP
php70u-common.x86_64 : Common files for PHP
php70u-dba.x86_64 : A database abstraction layer module for PHP applications
php70u-dbg.x86_64 : The interactive PHP debugger
php70u-devel.x86_64 : Files needed for building PHP extensions
php70u-embedded.x86_64 : PHP library for embedding in applications
php70u-enchant.x86_64 : Enchant spelling extension for PHP applications
php70u-fpm.x86_64 : PHP FastCGI Process Manager
php70u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php70u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php70u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php70u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php70u-imap.x86_64 : A module for PHP applications that use IMAP
php70u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php70u-intl.x86_64 : Internationalization extension for PHP applications
php70u-ioncube-loader.x86_64 : IonCube Loader provides PHP Modules to read IonCube Encoded Files
php70u-json.x86_64 : JavaScript Object Notation extension for PHP
php70u-ldap.x86_64 : A module for PHP applications that use LDAP
php70u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php70u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php70u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php70u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php70u-opcache.x86_64 : The Zend OPcache
php70u-pdo.x86_64 : A database access abstraction module for PHP applications
php70u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php70u-pear.noarch : PHP Extension and Application Repository framework
php70u-pecl-amqp.x86_64 : Communicate with any AMQP compliant server
php70u-pecl-apcu.x86_64 : APC User Cache
php70u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php70u-pecl-apcu-panel.noarch : APCu control panel
php70u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php70u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php70u-pecl-imagick.x86_64 : Provides a wrapper to the ImageMagick library
php70u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php70u-pecl-smbclient.x86_64 : PHP wrapper for libsmbclient
php70u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php70u-pgsql.x86_64 : A PostgreSQL database module for PHP
php70u-process.x86_64 : Modules for PHP script using system process interfaces
php70u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php70u-recode.x86_64 : A module for PHP applications for using the recode library
php70u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php70u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php70u-tidy.x86_64 : Standard PHP module provides tidy library support
php70u-xml.x86_64 : A module for PHP applications which use XML
php70u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
php71u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php71u-cli.x86_64 : Command-line interface for PHP
php71u-common.x86_64 : Common files for PHP
php71u-dba.x86_64 : A database abstraction layer module for PHP applications
php71u-dbg.x86_64 : The interactive PHP debugger
php71u-devel.x86_64 : Files needed for building PHP extensions
php71u-embedded.x86_64 : PHP library for embedding in applications
php71u-enchant.x86_64 : Enchant spelling extension for PHP applications
php71u-fpm.x86_64 : PHP FastCGI Process Manager
php71u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php71u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php71u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php71u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php71u-imap.x86_64 : A module for PHP applications that use IMAP
php71u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php71u-intl.x86_64 : Internationalization extension for PHP applications
php71u-json.x86_64 : JavaScript Object Notation extension for PHP
php71u-ldap.x86_64 : A module for PHP applications that use LDAP
php71u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php71u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php71u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php71u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php71u-opcache.x86_64 : The Zend OPcache
php71u-pdo.x86_64 : A database access abstraction module for PHP applications
php71u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php71u-pecl-apcu.x86_64 : APC User Cache
php71u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php71u-pecl-apcu-panel.noarch : APCu control panel
php71u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php71u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php71u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php71u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php71u-pgsql.x86_64 : A PostgreSQL database module for PHP
php71u-process.x86_64 : Modules for PHP script using system process interfaces
php71u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php71u-recode.x86_64 : A module for PHP applications for using the recode library
php71u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php71u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php71u-tidy.x86_64 : Standard PHP module provides tidy library support
php71u-xml.x86_64 : A module for PHP applications which use XML
php71u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
sclo-php70-php-pecl-apcu.x86_64 : APC User Cache
sclo-php70-php-pecl-apcu-bc.x86_64 : APCu Backwards Compatibility Module
sclo-php70-php-pecl-apcu-devel.x86_64 : APCu developer files (header)
sclo-php70-php-pecl-apfd.x86_64 : Always Populate Form Data
sclo-php70-php-pecl-http.x86_64 : Extended HTTP support
sclo-php70-php-pecl-http-devel.x86_64 : Extended HTTP support developer files (header)
sclo-php70-php-pecl-lzf.x86_64 : Extension to handle LZF de/compression
sclo-php70-php-pecl-mongodb.x86_64 : MongoDB driver for PHP
sclo-php70-php-pecl-propro.x86_64 : Property proxy
sclo-php70-php-pecl-raphf.x86_64 : Resource and persistent handles factory
sclo-php70-php-pecl-selinux.x86_64 : SELinux binding for PHP scripting language
sclo-php70-php-pecl-solr2.x86_64 : Object oriented API to Apache Solr
sclo-php70-php-pecl-uploadprogress.x86_64 : An extension to track progress of a file upload
sclo-php70-php-pecl-uuid.x86_64 : Universally Unique Identifier extension for PHP
sclo-php70-php-pecl-xattr.x86_64 : Extended attributes
sclo-php70-php-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
uwsgi-plugin-php70u.x86_64 : uWSGI - Plugin for PHP support

  Name and summary matches only, use "search all" for everything.

As we can see php7 is there. Great. But what about the php packages they have already? That’s coming up next.

Step 2. Check presence of plugin replace, we’ll use this to upgrade to mod_php70 once we’re ready

# Locate plugin replace is available
yum search yum-plugin-replace
# Install yum plugin replace if available (otherwise this will not work for you as easily)
yum install yum-plugin-replace

Step 3. Run a mock yum replace

# yum replace php53u --replace-with mod_php70u.x86_64
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
Error: No Package Matching mod_php70u.x86_64
[root@server3 ~]# yum replace php53u --replace-with mod_php70u
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00

WARNING: Unable to resolve all providers: ['config(php53u-common)', 'curl.so()(64bit)', 'fileinfo.so()(64bit)', 'json.so()(64bit)', 'phar.so()(64bit)', 'php-api', 'php-pecl(Fileinfo)', 'php-pecl(phar)', 'php-pecl(zip)', 'php-pecl-Fileinfo', 'php-pecl-phar', 'php-pecl-zip', 'php-zend-abi', 'php53(api)', 'php53(language)', 'php53(zend-abi)', 'php53-api', 'php53-bz2', 'php53-calendar', 'php53-common', 'php53-ctype', 'php53-curl', 'php53-date', 'php53-exif', 'php53-filter', 'php53-ftp', 'php53-gettext', 'php53-gmp', 'php53-hash', 'php53-iconv', 'php53-json', 'php53-libxml', 'php53-openssl', 'php53-pcre', 'php53-pecl(Fileinfo)', 'php53-pecl(json)', 'php53-pecl(phar)', 'php53-pecl(zip)', 'php53-pecl-Fileinfo', 'php53-pecl-json', 'php53-pecl-phar', 'php53-pecl-zip', 'php53-posix', 'php53-reflection', 'php53-session', 'php53-shmop', 'php53-simplexml', 'php53-sockets', 'php53-spl', 'php53-sqlite3', 'php53-sysvmsg', 'php53-sysvsem', 'php53-sysvshm', 'php53-tokenizer', 'php53-wddx', 'php53-zend-abi', 'php53-zip', 'php53-zlib', 'php53u(api)', 'php53u(language)', 'php53u(zend-abi)', 'php53u-api', 'php53u-bz2', 'php53u-calendar', 'php53u-ctype', 'php53u-curl', 'php53u-date', 'php53u-exif', 'php53u-fileinfo', 'php53u-filter', 'php53u-ftp', 'php53u-gettext', 'php53u-gmp', 'php53u-hash', 'php53u-iconv', 'php53u-json', 'php53u-libxml', 'php53u-openssl', 'php53u-pcre', 'php53u-pecl(Fileinfo)', 'php53u-pecl(json)', 'php53u-pecl(phar)', 'php53u-pecl(zip)', 'php53u-pecl-Fileinfo', 'php53u-pecl-json', 'php53u-pecl-phar', 'php53u-pecl-zip', 'php53u-posix', 'php53u-reflection', 'php53u-session', 'php53u-shmop', 'php53u-simplexml', 'php53u-sockets', 'php53u-spl', 'php53u-sqlite3', 'php53u-sysvmsg', 'php53u-sysvsem', 'php53u-sysvshm', 'php53u-tokenizer', 'php53u-wddx', 'php53u-zend-abi', 'php53u-zip', 'php53u-zlib', 'zip.so()(64bit)', 'php53u-common', 'php53u-common(x86-64)', 'php53-cgi', 'php53-cli', 'php53-pcntl', 'php53-readline', 'php53u-cgi', 'php53u-pcntl', 'php53u-readline', 'php53u-cli', 'php53u-cli(x86-64)', 'config(php53u-pdo)', 'pdo.so()(64bit)', 'pdo_sqlite.so()(64bit)', 'php53-pdo', 'php53-pdo-abi', 'php53-pdo_sqlite', 'php53u-pdo-abi', 'php53u-pdo', 'php53u-pdo(x86-64)', 'config(php53u-mysql)', 'mysql.so()(64bit)', 'mysqli.so()(64bit)', 'pdo_mysql.so()(64bit)', 'php-mysql', 'php53-mysql', 'php53-mysqli', 'php53u-mysqli', 'php53u-mysql', 'php53u-mysql(x86-64)', 'config(php53u)', 'libphp5.so()(64bit)', 'mod_php53u', 'php53', 'php53u', 'php53u(x86-64)', 'libphp5.so()(64bit)', 'php53-zts', 'php53u-zts', 'php53u-zts(x86-64)']

This may be normal depending on the package.  Continue? [y/N] y
Resolving Dependencies
--> Running transaction check
---> Package mod_php70u.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php53u.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-cli.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-common.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-mysql.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pdo.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pear.noarch 1:1.9.4-3.ius.centos6 will be erased
---> Package php53u-zts.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php70u-cli.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-common.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-gmp.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-json.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-mysqlnd.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pdo.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pear.noarch 1:1.10.1-2.ius.centos6 will be installed
---> Package php70u-process.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-xml.x86_64 0:7.0.14-3.ius.centos6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                        Version                                                                      Repository                                                 Size
==============================================================================================================================================================================================================================================================================
Installing:
 mod_php70u                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       2.7 M
 php70u-cli                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       4.0 M
 php70u-common                                                       x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       1.1 M
 php70u-gmp                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        65 k
 php70u-json                                                         x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        62 k
 php70u-mysqlnd                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       221 k
 php70u-pdo                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       115 k
 php70u-pear                                                         noarch                                                      1:1.10.1-2.ius.centos6                                                       ius                                                       362 k
 php70u-process                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        72 k
 php70u-xml                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       183 k
Removing:
 php53u                                                              x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.4 M
 php53u-cli                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      7.9 M
 php53u-common                                                       x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      3.4 M
 php53u-mysql                                                        x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      219 k
 php53u-pdo                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      126 k
 php53u-pear                                                         noarch                                                      1:1.9.4-3.ius.centos6                                                        @ius                                                      2.2 M
 php53u-zts                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.6 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install      10 Package(s)
Remove        7 Package(s)

Total download size: 8.8 M
Is this ok [y/N]: N
Exiting on user Command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx-2017-01-13-10-57L3T7JK.yumtx
You have mail in /var/spool/mail/root

Naturally, if you are satisfied that you do not need php53u-zts, the only php module which is not supported by PHP7, then you can proceed.

If you are wondering what ZTS is, The php-zts package contains a module for use with the Apache HTTP
Server which can operate under a threaded server processing model. (source pbone.net CentOS REPO)

ZTS is not required for MPM prefork, and is generally only used with MPM worker, afaik. So as long as your using prefork apache httpd your fine;

# apachectl -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

In our case prefork is being used, not worker. So I don’t think ZTS being missing is going to affect us. So we can proceed with typing ‘y’.

And’ thats pretty much how you upgrade to php7, it’s really easy with Rackspace IUS.

Troubleshooting Akamai CDN using Pragma Headers

Posted on 12th January 2017 by azio

Rackspace Cloud Files CDN enabled containers and the Rackspace CDN product can occasionally have an issue, in such cases, you can troubleshoot this a lot easier by using the DEBUG headers. To do this add -D and the folowing -H headers for your output

curl -I http://rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com/common/test/generator.png -D - -H "Pragma: akamai-x-get-client-ip, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no, akamai-x-feo-trace, akamai-x-get-request-id" -L

Naturally for this to work, you need to make sure you have a test URL. You can get one from Cloud Files in the Rackspace Control panel, or from whatever origin is configured with the CDN. Just login to the origin, and find a file path like httpdocs/somefolder/somefile.png and then get the raxcdn.com URL from the Rackspace CDN product page, or the Rackspace Cloud Files ‘show all links’ page on the cog icon next to the cloud files container. And add the path behind your documentroot.

So for the CDN URL rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com

For the files on the origin in the documentroot of the website configured with the CDN just add the ‘local’ document path within httpdocs or your www folder!

i.e. rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com becomes rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com/somefolder/somefile.png

All About NOVA and Xen Tools in Rackspace Cloud – why can’t I connect to my Windows server?

Posted on 9th November 2016 by azio

Why can’t I connect to my Rackspace Windows cloud-server, you ask? 2 important questions.

1. Is it a new build?
2. Is it using a custom image (a non rackspace base image).

(because the rackspace base images all have correct nova-agent and xen tools, so get networking information OK. But customer images don’t!). In the case you have run the below tests to see if nova-agent is running (or installed), you will need to install them.

Checking for the nova-agent and xe-guest-utilities

ps auxfwww | grep nova-agent
yum -qa xe-guest-utilities nova-agent
dpkg -l xe-guest-utilities nova-agent

Explanation and solution

Thanks for reaching out to us with your inquiry today. I’m glad to convey to you that I understand what the problem is with your cloud-server not being contactable.

Main reasons for breakage

The main reason why this is not working is most likely caused by some important pieces of software being missing. There is a piece of software called nova-agent, which is responsible for setting your cloud-servers IPV4 address, network subnet/mask, and ip routes, when it is first built. This is important, since the server image you built the server from, has different network details.

The rackspace build process giving networking detail to the VM is completely dependent on xe-guest-utilities and nova-agent

What has happened in this case, because the nova-agent wasn’t running on the cloud-server, the hypervisor software Rackspace use to automate cloud-server builds wasn’t able to contact the nova-agent running on your cloud-server, and therefore nova-agent wasn’t able to update the networking information. And hence, your not able to connect to it on it’s IPv4 address you are given at build time.

The steps to resolution: installing nova-agent and xen guest utilities
As such, nova-agent needs to be installed on the cloud-server you take the image from, it can be installed as follows:

https://community.rackspace.com/products/f/25/t/5694

Also nova-agent uses another piece of important software called xe-guest-utilities, or (Xen Tools) for your windows servers, this is an important ‘PV’ paravirtualization tools, responsible for seamless management of cloud-servers. Sorry that in this case it’s not working out seamlessly, but this can happen with images taken of servers which have had nova-agent disabled, uninstalled, or similar.

Upgrading the tools that nova-agent depends upon, can be installed by following the instructions at the following location:

https://support.rackspace.com/how-to/upgrade-citrix-xen-server-tools-for-windows-cloud-servers/

# Options of how to do this / Summary of Solution Steps

Naturally, you might be wondering how to achieve these changes, if you cannot RDP to the server. This is quite understandable, there are two ways to get this working;

Option 1) Manually install nova-agent on the current server you cannot access, then manually install the Xen Tools in the same way. This shall fix the OS on the server itself, and not the original image you built the server from. So it is important to create a new cloud-server image after performing these steps and us verifying tools + nova-agent installed correctly.

2) Manually install nova-agent on the source server you initially taken the image from, and install Xen Tools, then re-image the server, and then re-deploy. This should seamlesssly work each time on build with that image, provided the tools are installed. You will not need to recreate the image, since your fixing the problem on the cloud-server source that the original image was taken from.

I appreciate that these things are not 100% simple to get your head around and can be confusing for customers, I hope my explanation and summary makes this a little more painless to fix. Of course if you have additional questions, comments or concerns or don’t understand something I’ve said, please don’t hesitate to reach out to us, we are here to help!

Creating a proper Method of Retrieving, Sorting, and Parsing Rackspace CDN Access Logs

Posted on 7th November 2016 by azio

So, this has been rather a bane on the life which is lived as Adam Bull. Basically, a large customer of ours had 50+ CDN’s, and literally hundreds of gigabytes of Log Files. They were all in Rackspace Cloud Files, and the big question was ‘how do I know how busy my CDN is?’.

This is a remarkably good question, because actually, not many tools are provided here, and the customer will, much like on many other CDN services, have to download those logs, and then process them. But that is actually not easier either, and I spent a good few weeks (albeit when I had time), trying to figure out the best way to do this. I dabbled with using tree to display the most commonly used logs, I played with piwik, awstats, and many others such as goaccess, all to no avail, and even used a sophisticated AWK script from our good friends in Operations. No luck, nothing, do not pass go, or collect $200. So, I was actually forced to write something to try and achieve this, from start to finish. There are 3 problems.

1) how to easily obtain .CDN_ACCESS_LOGS from Rackspace Cloud Files to Cloud Server (or remote).
2) how to easily process these logs, in which format.
3) how to easily present these logs, using which application.

The first challenge was actually retrieving the files.

swiftly --verbose --eventlet --concurrency=100 get .CDN_ACCESS_LOGS --all-objects -o ./

Naturally to perform this step above, you will need a working, and setup swiftly environment. If you don’t know what swiftly, is or understand how to set up a swiftly envrionment, please see this article I wrote on the subject of deleting all files with swiftly (The howto explains the environment setup first! Just don’t follow the article to the end, and continue from here, once you’ve setup and installed swiftly)

Fore more info see:
https://community.rackspace.com/products/f/25/t/7190

Processing the Rackspace CDN Logs that we’ve downloaded, and organising them for further log processing
This required a lot more effort, and thought

The below script sits in the same folder as all of the containers

# ls -al 
total 196
drwxrwxr-x 36 root root  4096 Nov  7 12:33 .
drwxr-xr-x  6 root root  4096 Nov  7 12:06 ..
# used by my script
-rw-rw-r--  1 root root  1128 Nov  7 12:06 alldirs.txt

# CDN Log File containers as we downloaded them from swiftly Rackspace Cloud Files (.CDN_ACCESS_LOGS)
drwxrwxr-x  3 root root  4096 Oct 19 11:22 dev.demo.video.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:22 europe.assets.lon.tv
drwxrwxr-x  5 root root  4096 Oct 19 11:22 files.lon.cdn.lon.com
drwxrwxr-x  3 root root  4096 Oct 19 11:23 files.blah.cdn..com
drwxrwxr-x  5 root root  4096 Oct 19 11:24 files.demo.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:25 files.invesco.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:25 files.test.cdn..com
-rw-r--r--  1 root root   561 Nov  7 12:02 generate-report.sh
-rwxr-xr-x  1 root root  1414 Nov  7 12:15 logparser.sh

# Used by my script
drwxr-xr-x  2 root root  4096 Nov  7 12:06 parsed
drwxr-xr-x  2 root root  4096 Nov  7 12:33 parsed-combined

#!/bin/bash

# Author : Adam Bull
# Title: Rackspace CDN Log Parser
# Date: November 7th 2016

echo "Deleting previous jobs"
rm -rf parsed;
rm -rf parsed-combined

ls -ld */ | awk '{print $9}' | grep -v parsed > alldirs.txt


# Create Location for Combined File Listing for CDN LOGS
mkdir parsed

# Create Location for combined CDN or ACCESS LOGS
mkdir parsed-combined

# This just builds a list of the CDN Access Logs
echo "Building list of Downloaded .CDN_ACCESS_LOG Files"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
echo $folder
        echo "$m" | xargs -i find ./{} -type f -print > "parsed/$folder.log"
done < alldirs.txt

# This part cats the files and uses xargs to produce all the Log oiutput, before cut processing and redirecting to parsed-combined/$folder
echo "Combining .CDN_ACCESS_LOG Files for bulk processing and converting into NCSA format"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
cat "parsed/$folder.log" | xargs -i zcat {} | cut -d' ' -f1-10  > "parsed-combined/$folder"
done < alldirs.txt


# This part processes the Log files with Goaccess, generating HTML reports
echo "Generating Goaccess HTML Logs"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
goaccess -f "parsed-combined/$folder" -a -o "/var/www/html/$folder.html"
done < alldirs.txt

How to easily present these logs

I kind of deceived you with the last step. Actually, because I have already done it, with the above script. Though, you will naturally need to have an httpd installed, and a documentroot in /var/www/html, so make sure you install apache2:

yum install httpd awstats

De de de de de de da! da da!

Some little caveats:

Generating a master index.html file of all the sites

[root@cdn-log-parser-mother html]# pwd /var/www/html [root@cdn-log-parser-mother html]# ls -al | awk '{print $9}' | xargs -i echo " {} " > index.html

I will expand the script to generate this automatically soon, but for now, leaving like this due to time constraints.

Checking a Website or Rackspace Load Balancers Supported SSL Ciphers

Posted on 11th October 2016 by azio

So, you may have recently had an audit performed, or have been warned about the dangers of SSLv3, poodle attack, heartbleed and etc. You want to understand exactly which ciphers your using on the Load Balancer, cloud-server, or dedicated server. It’s actually very easy to do this with nmap. Install it first, naturally.

# CentOS / RedHat
yum install nmap

# Debian / Ubuntu
apt-get install nmap

# Check for SSL ciphers

# nmap hostnamegoeshere.com --script ssl-enum-ciphers -p 443

Nmap done: 1 IP address (1 host up) scanned in 1.57 seconds

In this case we can see that only TLS v1.1 and TLS v1.0 are supported. No TLSv1.2 and no SSLv3.

Cheers &
Best wishes,
Adam

Using Sar to tell a story

Posted on 12th August 2016 by azio

So, a customer is experiencing slowness/sluggishness in their app. You know there is not issue with the hypervisor from instinct, but instinct isn’t enough. Using tools like xentop, sar, bwm-ng are critical parts of live and historical troubleshooting.

Sar can tell you a story, if you can ask the storyteller the write questions, or even better, pick up the book and read it properly. You’ll understand what the plot, scenario, situation and exactly how to proceed with troubleshooting by paying attention to these data and knowing which things to check under certain circumstances.

This article doesn’t go in depth to that, but it gives you a good reference of a variety of tests, the most important being, cpu usage, io usage, network usage, and load averages.

CPU Usage of all processors

# Grab details live
sar -u 1 3

# Use historical binary sar file
# sa10 means '10th day' of current month.
sar -u -f /var/log/sa/sa10

CPU Usage of a particular Processor

sar -P ALL 1 1

‘-P 1’ means check only the 2nd Core. (Core numbers start from 0).

sar -P 1 1 5

The above command displays real time CPU usage for core number 1, every 1 second for 5 times.

Observing Changes in Memory over time

sar -r 1 3

The above command provides memory stats every 1 second for a total of 3 times.

Observing Swap usage over time

sar -S 1 5

The above command reports swap statistics every 1 seconds, a total 3 times.

Overall I/O activity

sar -b 1 3

The above command checks every 1 seconds, 3 times.

Individual Block Device I/O Activities

This is a useful check for LUN , block devices and other specific mounts

sar -d 1 1 
sar -p d

DEV – indicates block device, i.e. sda, sda1, sdb1 etc.

Total Number processors created a second / Context switches

sar -w 1 3

Run Queue and Load Average

sar -q 1 3

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3” reports for every 1 seconds a total of 3 times.

Report Network Statistics

sar -n KEYWORD

KEYWORDS Available;

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information. The output will be very long.

sar -n DEV 1 1

Specify Start Time

sar -q -f /var/log/sa/sa11 -s 11:00:00

sar -q -f /var/log/sa/sa11 -s 11:00:00 | head -n 10

Using TCP to ping test your Cloud server connectivity

Posted on 10th August 2016 by azio

So, you have probably heard that there are a variety of reasons why you shouldn’t use ICMP to test your service is operating normally. Mainly because of the way that ICMP is handled by routers. If you really want a representative view of the way that TCP packets, such as HTTP and HTTPS are performing in terms of packet loss (that is to say packets which do not arrive at their destination) , then hping is your friend.

You might be pinging a cloud-server that is not replying. You might think it’s down. But what if the firewall is simply dropping ICMP echo requests coming in on that port? Indeed.

Enter hping.

# hping -S -p 80 google.com
HPING google.com (eth0 74.125.136.102): S set, 40 headers + 0 data bytes
len=46 ip=74.125.136.102 ttl=46 id=23970 sport=80 flags=SA seq=0 win=42780 rtt=13.8 ms
len=46 ip=74.125.136.102 ttl=47 id=37443 sport=80 flags=SA seq=1 win=42780 rtt=12.6 ms
len=46 ip=74.125.136.102 ttl=47 id=43654 sport=80 flags=SA seq=2 win=42780 rtt=12.0 ms
len=46 ip=74.125.136.102 ttl=47 id=37877 sport=80 flags=SA seq=3 win=42780 rtt=11.4 ms
len=46 ip=74.125.136.102 ttl=47 id=62433 sport=80 flags=SA seq=4 win=42780 rtt=13.3 ms
^C
--- google.com hping statistic ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 11.4/12.6/13.8 ms

In this case I tested with google.com. I’m actually surprised that more people don’t use hping, because, hping is awesome. It also makes quite a decent port scanner, were it not for the fact that the machine I tried to test that feature with buffer overflowed 😉 It’s a nice way to test a firewalled box, but more than that, it’s a more reliable test in my opinion.

haxed.me.uk

a place of infrastructure delights

Category Archives: Interweb