Category Archives: Uncategorised

How I fixed a really broken CentOS Machine after an dist upgrade

Posted on by

So… you do basic dist upgrade.. you don’t really make an effort to take a cloud-server image (because your me, and are an idiot! and dont even follow the advice you give your own customers, yourself!)…

Now I’ve got that bit over.. I basically found that udev wasn’t installed. This might not seem like completely the end of the world, however, the fact that /dev/random /dev/urandom were missing, and /dev/null was a regular file.. we had big issues man.. I didn’t think I’d be able to get SSH to behave again, but actually it seems this is isolated to udev in most latest centos revisions.

I’m not going to give my kernel version, but you get the idea ;-D

[root@RESCUE-pirax-test ~]# ls -al
total 40
dr-xr-x---.  5 root root 4096 Dec  7 09:34 .
dr-xr-xr-x. 18 root root 4096 Dec  7 09:34 ..
-rw-r--r--.  1 root root   18 Dec 29  2013 .bash_logout
-rw-r--r--.  1 root root  176 Dec 29  2013 .bash_profile
-rw-r--r--.  1 root root  176 Dec 29  2013 .bashrc
drwxr-xr-x.  3 root root 4096 Dec  7 09:34 .cache
drwxr-xr-x.  3 root root 4096 Dec  7 09:34 .config
-rw-r--r--.  1 root root  100 Dec 29  2013 .cshrc
drwx------.  2 root root 4096 Dec  7 09:34 .ssh
-rw-r--r--.  1 root root  129 Dec 29  2013 .tcshrc
[root@RESCUE-pirax-test ~]# cd /mnt
[root@RESCUE-pirax-test mnt]# ls -al
total 112
dr-xr-xr-x. 23 root root  4096 Dec  7 09:32 .
dr-xr-xr-x. 18 root root  4096 Dec  7 09:34 ..
lrwxrwxrwx.  1 root root     7 Feb 23  2016 bin -> usr/bin
dr-xr-xr-x.  4 root root  4096 Dec  6 11:53 boot
drwxr-xr-x.  2 root root  4096 Apr 21  2016 customer
drwxr-xr-x.  2 root root  4096 Dec  7 09:47 dev
drwxr-xr-x.  8 root root  4096 Jun 10 10:02 documents
drwxr-xr-x.  3 root root  4096 Mar  1  2016 dump
drwxr-xr-x. 91 root root 12288 Dec  7 09:45 etc
drwxr-xr-x.  5 root root  4096 Nov 11 11:58 home
drwxr-xr-x.  3 root root  4096 Nov  9  2015 include
lrwxrwxrwx.  1 root root     7 Feb 23  2016 lib -> usr/lib
lrwxrwxrwx.  1 root root     9 Feb 23  2016 lib64 -> usr/lib64
drwx------.  2 root root 16384 Sep  3  2015 lost+found
drwxr-xr-x.  2 root root  4096 Aug 12  2015 media
drwxr-xr-x.  3 root root  4096 Jun 23 14:23 mnt
drwxr-xr-x.  4 root root  4096 Aug 12  2015 opt
drwxr-xr-x.  2 root root  4096 Sep  3  2015 proc
drwxr-xr-x. 26 root root  4096 Dec  7 09:36 root
drwxr-xr-x.  3 root root  4096 Dec  7 09:43 run
lrwxrwxrwx.  1 root root     8 Feb 23  2016 sbin -> usr/sbin
drwxr-xr-x.  2 root root  4096 Aug 12  2015 srv
drwxr-xr-x.  2 root root  4096 Sep  3  2015 sys
drwxrwxrwt.  7 root root  4096 Dec  7 09:46 tmp
drwxr-xr-x. 13 root root  4096 Feb 23  2016 usr
drwxr-xr-x. 22 root root  4096 Dec  7 09:31 var
drwxr-xr-x.  9 root root  4096 Oct 12  2015 wpscan
[root@RESCUE-pirax-test mnt]# cd ..
[root@RESCUE-pirax-test /]# chroot /mnt
[root@RESCUE-pirax-test /]# ls -al
total 112
dr-xr-xr-x. 23 root root  4096 Dec  7 09:32 .
dr-xr-xr-x. 23 root root  4096 Dec  7 09:32 ..
lrwxrwxrwx.  1 root root     7 Feb 23  2016 bin -> usr/bin
dr-xr-xr-x.  4 root root  4096 Dec  6 11:53 boot
drwxr-xr-x.  2 root root  4096 Apr 21  2016 customer
drwxr-xr-x.  2 root root  4096 Dec  7 09:47 dev
drwxr-xr-x.  8 root root  4096 Jun 10 10:02 documents
drwxr-xr-x.  3 root root  4096 Mar  1  2016 dump
drwxr-xr-x. 91 root root 12288 Dec  7 09:45 etc
drwxr-xr-x.  5 root root  4096 Nov 11 11:58 home
drwxr-xr-x.  3 root root  4096 Nov  9  2015 include
lrwxrwxrwx.  1 root root     7 Feb 23  2016 lib -> usr/lib
lrwxrwxrwx.  1 root root     9 Feb 23  2016 lib64 -> usr/lib64
drwx------.  2 root root 16384 Sep  3  2015 lost+found
drwxr-xr-x.  2 root root  4096 Aug 12  2015 media
drwxr-xr-x.  3 root root  4096 Jun 23 14:23 mnt
drwxr-xr-x.  4 root root  4096 Aug 12  2015 opt
drwxr-xr-x.  2 root root  4096 Sep  3  2015 proc
drwxr-xr-x. 26 root root  4096 Dec  7 09:36 root
drwxr-xr-x.  3 root root  4096 Dec  7 09:43 run
lrwxrwxrwx.  1 root root     8 Feb 23  2016 sbin -> usr/sbin
drwxr-xr-x.  2 root root  4096 Aug 12  2015 srv
drwxr-xr-x.  2 root root  4096 Sep  3  2015 sys
drwxrwxrwt.  7 root root  4096 Dec  7 09:46 tmp
drwxr-xr-x. 13 root root  4096 Feb 23  2016 usr
drwxr-xr-x. 22 root root  4096 Dec  7 09:31 var
drwxr-xr-x.  9 root root  4096 Oct 12  2015 wpscan
[root@RESCUE-pirax-test /]# su adam
[adam@RESCUE-pirax-test /]$ ssh root@localhost
cannot read from /dev/urandom, No such file or directory
[adam@RESCUE-pirax-test /]$ yum update mkinitrd
Loaded plugins: fastestmirror, langpacks
You need to be root to perform this command.
[adam@RESCUE-pirax-test /]$ exit
exit
[root@RESCUE-pirax-test /]# yum update mkinitrd
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.vooservers.com
 * epel: epel.check-update.co.uk
 * extras: mirrors.vooservers.com
 * updates: mirrors.vooservers.com
No Match for argument: mkinitrd
No package mkinitrd available.
No packages marked for update
[root@RESCUE-pirax-test /]# yum provides initrd
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.vooservers.com
 * epel: epel.check-update.co.uk
 * extras: mirrors.vooservers.com
 * updates: mirrors.vooservers.com
No matches found
[root@RESCUE-pirax-test /]# cd /dev
[root@RESCUE-pirax-test dev]# /sbin/MAKEDEV urandom;
bash: /sbin/MAKEDEV: No such file or directory
[root@RESCUE-pirax-test dev]# yum install udev
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.vooservers.com
 * epel: epel.check-update.co.uk
 * extras: mirrors.vooservers.com
 * updates: mirrors.vooservers.com
Resolving Dependencies
--> Running transaction check
---> Package systemd.x86_64 0:219-19.el7_2.4 will be updated
--> Processing Dependency: systemd = 219-19.el7_2.4 for package: systemd-python-219-19.el7_2.4.x86_64
--> Processing Dependency: systemd = 219-19.el7_2.4 for package: systemd-sysv-219-19.el7_2.4.x86_64
---> Package systemd.x86_64 0:219-19.el7_2.13 will be an update
--> Processing Dependency: systemd-libs = 219-19.el7_2.13 for package: systemd-219-19.el7_2.13.x86_64
--> Running transaction check
---> Package systemd-libs.x86_64 0:219-19.el7_2.4 will be updated
--> Processing Dependency: systemd-libs = 219-19.el7_2.4 for package: libgudev1-219-19.el7_2.4.x86_64
---> Package systemd-libs.x86_64 0:219-19.el7_2.13 will be an update
---> Package systemd-python.x86_64 0:219-19.el7_2.4 will be updated
---> Package systemd-python.x86_64 0:219-19.el7_2.13 will be an update
---> Package systemd-sysv.x86_64 0:219-19.el7_2.4 will be updated
---> Package systemd-sysv.x86_64 0:219-19.el7_2.13 will be an update
--> Running transaction check
---> Package libgudev1.x86_64 0:219-19.el7_2.4 will be updated
---> Package libgudev1.x86_64 0:219-19.el7_2.13 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                        Version                                                              Repository                                                    Size
=========================================================================================================================================================================================================================================================================
Updating:
 systemd                                                             x86_64                                                      219-19.el7_2.13                                                      updates                                                      5.1 M
Updating for dependencies:
 libgudev1                                                           x86_64                                                      219-19.el7_2.13                                                      updates                                                       67 k
 systemd-libs                                                        x86_64                                                      219-19.el7_2.13                                                      updates                                                      358 k
 systemd-python                                                      x86_64                                                      219-19.el7_2.13                                                      updates                                                      100 k
 systemd-sysv                                                        x86_64                                                      219-19.el7_2.13                                                      updates                                                       54 k

Transaction Summary
=========================================================================================================================================================================================================================================================================
Upgrade  1 Package (+4 Dependent packages)

Total size: 5.7 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : systemd-libs-219-19.el7_2.13.x86_64                                                                                                                                                                                                                  1/10
  Updating   : systemd-219-19.el7_2.13.x86_64                                                                                                                                                                                                                       2/10
  Updating   : systemd-sysv-219-19.el7_2.13.x86_64                                                                                                                                                                                                                  3/10
  Updating   : systemd-python-219-19.el7_2.13.x86_64                                                                                                                                                                                                                4/10
  Updating   : libgudev1-219-19.el7_2.13.x86_64                                                                                                                                                                                                                     5/10
  Cleanup    : systemd-sysv-219-19.el7_2.4.x86_64                                                                                                                                                                                                                   6/10
  Cleanup    : systemd-python-219-19.el7_2.4.x86_64                                                                                                                                                                                                                 7/10
  Cleanup    : systemd-219-19.el7_2.4.x86_64                                                                                                                                                                                                                        8/10
  Cleanup    : libgudev1-219-19.el7_2.4.x86_64                                                                                                                                                                                                                      9/10
  Cleanup    : systemd-libs-219-19.el7_2.4.x86_64                                                                                                                                                                                                                  10/10
  Verifying  : systemd-libs-219-19.el7_2.13.x86_64                                                                                                                                                                                                                  1/10
  Verifying  : systemd-sysv-219-19.el7_2.13.x86_64                                                                                                                                                                                                                  2/10
  Verifying  : systemd-219-19.el7_2.13.x86_64                                                                                                                                                                                                                       3/10
  Verifying  : systemd-python-219-19.el7_2.13.x86_64                                                                                                                                                                                                                4/10
  Verifying  : libgudev1-219-19.el7_2.13.x86_64                                                                                                                                                                                                                     5/10
  Verifying  : systemd-libs-219-19.el7_2.4.x86_64                                                                                                                                                                                                                   6/10
  Verifying  : systemd-sysv-219-19.el7_2.4.x86_64                                                                                                                                                                                                                   7/10
  Verifying  : systemd-219-19.el7_2.4.x86_64                                                                                                                                                                                                                        8/10
  Verifying  : libgudev1-219-19.el7_2.4.x86_64                                                                                                                                                                                                                      9/10
  Verifying  : systemd-python-219-19.el7_2.4.x86_64                                                                                                                                                                                                                10/10

Updated:
  systemd.x86_64 0:219-19.el7_2.13

Dependency Updated:
  libgudev1.x86_64 0:219-19.el7_2.13                             systemd-libs.x86_64 0:219-19.el7_2.13                             systemd-python.x86_64 0:219-19.el7_2.13                             systemd-sysv.x86_64 0:219-19.el7_2.13

Complete!
[root@RESCUE-pirax-test dev]# su adam
[adam@RESCUE-pirax-test dev]$ ssh root@localhost
cannot read from /dev/urandom, No such file or directory
[adam@RESCUE-pirax-test dev]$ /sbin/
Display all 526 possibilities? (y or n)
[adam@RESCUE-pirax-test dev]$ /sbin/MAKEDEV std
bash: /sbin/MAKEDEV: No such file or directory
[adam@RESCUE-pirax-test dev]$ mknod /dev/random c 1 9
mknod: ‘/dev/random’: Permission denied
[adam@RESCUE-pirax-test dev]$ exit
exit
[root@RESCUE-pirax-test dev]# mknod /dev/random c 1 9
[root@RESCUE-pirax-test dev]# su adam
[adam@RESCUE-pirax-test dev]$ ssh root@localhost
cannot read from /dev/urandom, No such file or directory
[adam@RESCUE-pirax-test dev]$ exit
exit
[root@RESCUE-pirax-test dev]# mknod /dev/urandom c 1 9
[root@RESCUE-pirax-test dev]# su adam
[adam@RESCUE-pirax-test dev]$ ssh root@localhost
Host key verification failed.
[adam@RESCUE-pirax-test dev]$ exit



Dec  7 09:23:55 pirax-test login: FAILED LOGIN 1 FROM tty1 FOR root, Authentication failure
Dec  7 09:32:00 pirax-test polkitd[1031]: Loading rules from directory /etc/polkit-1/rules.d
Dec  7 09:32:00 pirax-test polkitd[1031]: Loading rules from directory /usr/share/polkit-1/rules.d
Dec  7 09:32:00 pirax-test polkitd[1031]: Finished loading, compiling and executing 2 rules
Dec  7 09:32:00 pirax-test polkitd[1031]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
Dec  7 09:32:10 pirax-test sshd[1375]: Server listening on 0.0.0.0 port 666.
Dec  7 09:32:10 pirax-test sshd[1375]: Server listening on :: port 666.
Dec  7 09:32:24 pirax-test unix_chkpwd[2692]: password check failed for user (root)
Dec  7 09:32:24 pirax-test login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=  user=root
Dec  7 09:32:24 pirax-test login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Dec  7 09:32:27 pirax-test login: FAILED LOGIN 1 FROM tty1 FOR root, Authentication failure
Dec  7 09:32:32 pirax-test unix_chkpwd[2694]: password check failed for user (root)
Dec  7 09:32:32 pirax-test login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Dec  7 09:32:34 pirax-test login: FAILED LOGIN 2 FROM tty1 FOR root, Authentication failure
Dec  7 09:32:39 pirax-test unix_chkpwd[2696]: password check failed for user (root)
Dec  7 09:32:39 pirax-test login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Dec  7 09:32:41 pirax-test login: FAILED LOGIN SESSION FROM tty1 FOR root, Authentication failure
Dec  7 09:32:41 pirax-test login: PAM 2 more authentication failures; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=  user=root
Dec  7 10:05:17 pirax-test polkitd[1029]: Loading rules from directory /etc/polkit-1/rules.d
Dec  7 10:05:17 pirax-test polkitd[1029]: Loading rules from directory /usr/share/polkit-1/rules.d
Dec  7 10:05:17 pirax-test polkitd[1029]: Finished loading, compiling and executing 2 rules
Dec  7 10:05:17 pirax-test polkitd[1029]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
Dec  7 10:05:27 pirax-test sshd[1375]: Server listening on 0.0.0.0 port 666.
Dec  7 10:05:27 pirax-test sshd[1375]: Server listening on :: port 666.
Dec  7 10:09:56 pirax-test unix_chkpwd[2791]: password check failed for user (root)
Dec  7 10:09:56 pirax-test login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=  user=root
Dec  7 10:09:56 pirax-test login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Dec  7 10:09:58 pirax-test login: FAILED LOGIN 1 FROM tty1 FOR root, Authentication failure
Dec  7 10:10:07 pirax-test unix_chkpwd[2802]: password check failed for user (root)
Dec  7 10:10:07 pirax-test login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Dec  7 10:10:09 pirax-test login: FAILED LOGIN 2 FROM tty1 FOR root, Authentication failure
[root@RESCUE-pirax-test /]# adduser adam
adduser: user 'adam' already exists
[root@RESCUE-pirax-test /]# passwd adam
Changing password for user adam.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@RESCUE-pirax-test /]# passwd root
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@RESCUE-pirax-test /]# su adam
[adam@RESCUE-pirax-test /]$ ssh adam@localhost
Host key verification failed.
[adam@RESCUE-pirax-test /]$ exit
exit
[root@RESCUE-pirax-test /]# vi /etc/ss
ssh/ ssl/
[root@RESCUE-pirax-test /]# vi /etc/ss
ssh/ ssl/
[root@RESCUE-pirax-test /]# vi /etc/ssh/
moduli                    ssh_config                sshd_config               ssh_host_ecdsa_key        ssh_host_ecdsa_key.pub    ssh_host_ed25519_key      ssh_host_ed25519_key.pub  ssh_host_rsa_key          ssh_host_rsa_key.pub
[root@RESCUE-pirax-test /]# vi /etc/ssh/sshd_config
[root@RESCUE-pirax-test /]# su adam
[adam@RESCUE-pirax-test /]$ ssh adam@localhost
Host key verification failed.
[adam@RESCUE-pirax-test /]$ exit
exit
[root@RESCUE-pirax-test /]# mknod -m 666 /dev/tty c 5 0
[root@RESCUE-pirax-test /]# su adam
[adam@RESCUE-pirax-test /]$ ssh adam@localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is e1:0c:0f:95:94:55:84:00:0d:e9:36:42:1d:6d:9e:0c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
adam@localhost's password:

It’s still kind of broken, but its getting less broken by the minute ;-D

Rackspace Cloud Server not coming up after building it

Posted on by

So! Perhaps you’ve read my article on nova-agent , the common cause of this issue? If you haven’t you should since it covers out well the importance of nova-agent.

However, nova-agent itself also comes unstuck if the machine nova-agent is installed on is missing the xe-linux-distribution service, this service is provided by the package xe-guest-utilities and can be installed by yourselves, in the case that installing/ ensuring nova-agent starts on boot does not fix your issues.

Specifically if your nova-agent log provides you this message, you know you need to install the xe-guest-utilities. Simplies!

Problem 

# cat /var/log/nova-agent.log

2016-10-06 18:58:14,696 [ERROR] [EXC] Traceback (most recent call last):
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/nova-agent.py", line 40, in 
2016-10-06 18:58:14,697 [ERROR] [EXC]     xs = plugins.XSComm()
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/1.39.0/plugins/xscomm.py", line 43, in __init__
2016-10-06 18:58:14,697 [ERROR] [EXC]     self.xs_handle = pyxenstore.Handle()
2016-10-06 18:58:14,700 [ERROR] [EXC] PyXenStoreError: Couldn't open connection to the xenstore: No such file or directory
2016-10-06 18:58:14,701 [ERROR] failed to parse config file '/usr/share/nova-agent/nova-agent.py'

Solution 

# Redhat and CentOS systems
yum install xe-guest-utilities

# Debian, Ubuntu and other apt based systems
apt-get install xe-guest-utilities

I hope that this is of some assistance, here is some more background information.

More details about nova-agent and xe-guest-utilities in Xen

Provided that you have definitely enabled nova-agent, and ensured that it is running (after restarting the original server), with ps auxfwww | grep nova-agent

then, you should be good to re-image the original server, and then rebuild out the second.

The reason why your server doesn’t appear to be coming up in the new build is for some reason, nova-agent service got disabled on boot-time, and as a result, the nova-agent service responsible for swapping out the network configuration of your cloud-server wasn’t started up when the server was built, and the automatic ip configuration change didn’t occur. This explains well the behaviour you’ve been seeing, and after looking in the backend the error code seems to confirm that the issue was that the nova-agent wasn’t running.

Provided that you’ve definitely installed nova-agent and confirmed it is running, as well as made sure it starts at boot time, as in the article I wrote, you should be good.

I hope that this explanation and clarification meets you well.

I can see that you’ve recently posted an additional issue that has been experienced with xe-linux-distribution (the cause of the PyXenStoreError). This secondary cause of the issue can be fixed by ensuring xe-linux-distribution is installed;

apt-get update;
apt-get install xe-guest-utilities

This should install the xen guest tools as required by nova-agent. This is required by the nova-agent in order for the networking data to be retrieved by nova-agent, whereas nova-agent itself applies the change, but these services both need to be running and installed for this to work properly!

I really hope that this is of some assistance ,of course if you have additional questions, comments or concerns please don’t hesitate to write back, and we can escalate this issue further for you. These instructions should fix your issues though! I hope this helps &

Cool Little script for downloading stuff

Posted on by 

#!/bin/sh
# just use uuid's instead of sequential numbers hehe

for a in `seq 10000000 90000000`;
do
for b in `seq 1 10`;
do
‪#‎echo‬ http://cdn.anonymous.com/$a""_""$b"".user"
echo "wget http://cdn.anonymous.com/$a""_""$b"".user" | bash
#echo "curl http://cdn.anonymous.com/$a""_""$b".user -o "($a)"_"($b)".user"

filesize=`ls -al "$a"_"$b".user | awk '{print $5}'`
echo "FILESIZE= $filesize"

if [ "$filesize" -eq "49" ]
then
echo "404: Emtpy fakefile HTTP 200 detected! The end of this hidden usergroup was detected"
echo "Cleaning up.."
rm "$a"_"$b".user
break;

else
echo "200: Continuing "
fi

sleep 4

done

TCPDUMP command packet capture Usage

Posted on by

So, it’s been a little while since my last update. We’ve been quite busy recently, but for those interested in learning more about tcpdump and physically capturing packets.

List Interfaces that can be tcp dumped 

tcpdump -D

Listen on Interface eth0

tcpdump -i eth0

Listen to Xenserver domain 16 on public net 

tcpdump -i vif16.0

Listen on any interface 

tcpdump -i any

Super duper High verbosity tcpdump 

tcpdump -vvvv -i eth0

Be verbose and print data of each packet in both hex and ASCII 

tcpdump -v -X -i eth0

Be less verbose 

tcpdump -q

Limit the capture of packets to 100 

tcpdump -c 100 -i eth0

Display IP addresses and port numbers instead of domain and service names when capturing packets (note: on some systems you need to specify -nn to display port numbers):

tcpdump -n

Capture any packets where the destination host is 192.168.1.1. Display IP addresses and port numbers: 

tcpdump -n dst host 192.168.1.1

Capture any packets where the source host is 192.168.1.1. Display IP addresses and port numbers: 

tcpdump -n src host 192.168.1.1

http://www.rationallyparanoid.com/articles/tcpdump.html

Finding Stuff quick and dirty way

Posted on by

Hey. So my good friend who is a support engineer was asking me how he could find mail log that wasn’t in the traditional location and he was scratching his head.
So I put this together (which by the way is really bad), but not in a harmful way, it could just be more elegant. But since he is still learning , this seemed like a good time to introduce him to xargs.

find / | grep mail | grep log | xargs -i ls -al {}

Nice and simple though and pretty much straight to the point, if the grep pipes are forgiven. (and wouldn’t blame you if they were not 🙂 )

Building 50 Cloud Servers BASH/API Automation

Posted on by

So, I had a good friend of mine who is a cloud Mentor at Rackspace, reach out to me concerning an easier way of deploying cloud-images, without each time a cloud server is spun up, having to unroll the image into a CBS. His customer wanted to simply have a ‘primary master’ CBS volume, a template of their site if you will, the equivalent of a ‘golden image’, the only difference was it was a CBS Volume. So I set about making this work. It would still take a few hours, at least to provision 50 to 200 servers, but it was much faster than the alternatives. Here is how I did it. I actually have some ideas for how to improve this but I’ve not yet implemented it. That goody is to come in later scripts.

#!/bin/bash

USERNAME='mycloudusername'
APIKEY='mycloudapikey'
ACCOUNT_NUMBER=100101011
API_ENDPOINT="https://lon.blockstorage.api.rackspacecloud.com/v1/$ACCOUNT_NUMBER/volumes"
MASTER_CBS_VOL_ID="d8a67ad1-8037-46bc-8790-efca2cb6e5bd"


TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`


# Populate CBS
for i in `seq 1 2`;
do

echo "Generating CBS Clone #$i"
#curl -s -vvvv  \
-X POST "$API_ENDPOINT" \
-H "X-Auth-Token: $TOKEN"  \
-H "X-Project-Id: $ACCOUNT_NUMBER" \
-H "Accept: application/json"  \
-H "Content-Type: application/json" -d '{"volume": {"source_volid": "d8a67ad1-8037-46bc-8790-efca2cb6e5bd", "size": 50, "display_name": "win-'$i'", "volume_type": "SSD"}}'  | jq .volume.id | tr -d '"' >> cbs.created

done

echo "Giving CBS 2 hour grace time for 50 CBS clone"
#sleep 7200

echo "Listing all CBS Volume ID's created"
cat cbs.created
echo ""


# Populate Nova
count=1;
echo "Populating Nova servers with CBS disk"
while read n; do
Echo "Build Task $n Started:"
nova --insecure --os-username mycloudusername --os-auth-system=rackspace  --os-tenant-name 100110111 --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password myapikeygoeshere boot --flavor general1-1 --block-device-mapping vda="$n":::1 Auto-win-"$count"
((count=count+1))

done < cbs.created

# Move the cbs.created.old away
mv cbs.created cbs.created.old -f

Requirements are nova and jq.
https://stedolan.github.io/jq/
https://developer.rackspace.com/blog/getting-started-using-python-novaclient-to-manage-cloud-servers/

Checking for Network packet Retransmission , troubleshooting network card & switches

Posted on by

So, you might want to test whether the NIC of your box is ‘bad’, one way to do this is looking at the retransmissions.

netstat -s | grep retransmits
   3535665 fast retransmits
   3920918 forward retransmits
   122319 retransmits in slow start
   3652 sack retransmits failed

netstat -s | grep transmit
    10512472 segments retransmited
    733 times recovered from packet loss due to fast retransmit
    Detected reordering 73 times using reno fast retransmit
    TCPLostRetransmit: 196
    400 timeouts after reno fast retransmit
    3535665 fast retransmits
    3920918 forward retransmits
    122319 retransmits in slow start
    13652 sack retransmits failed

This isn’t much use though, because you need to see how many total packets come in:

netstat -s | grep total
    23799703342 total packets received

It’s possible to get the full details with netstat -s , naturally.

Checking Forward and Reverse Connectivity of a Linux Server

Posted on by

A good friend of mine is to thank for this excellent pair of one liners. One is to be executed on source, and the other on destination target.

Testing forward route 


# Machine Source
root@iup2-web01:/mnt/www# dd if=/dev/zero bs=1024K count=1024 | nc -v 10.181.164.100 23
Connection to 10.181.164.100 23 port [tcp/telnet] succeeded!
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.0009 s, 97.6 MB/s
---

# Machine Destination
root@iup2-nfs:~# nc -v -l 23 > /dev/null
Listening on [0.0.0.0] (family 0, port 23)
Connection from [10.181.162.15] port 23 [tcp/telnet] accepted (family 2, sport 54373)

However when we try this in reverse, we see a major degradation in the network speed. Here is test with 20MB transfer instead of 1000MB:

Testing Reverse Route 

---

# Machine Source
root@iup2-nfs:~# dd if=/dev/zero bs=1024K count=20 | nc -v 10.181.162.15 23
Connection to 10.181.162.15 23 port [tcp/telnet] succeeded!
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 144.327 s, 145 kB/s
---
# Machine Destination
root@iup2-web01:/mnt/www#  nc -v -l 23 > /dev/null 
Listening on [0.0.0.0] (family 0, port 23)
Connection from [10.181.164.100] port 23 [tcp/telnet] accepted (family 2, sport 56072)

As we can see one of the machines has some difficulty. The issue at hand was that there was some problems with the virtual switch daemon on the hypervisor. Thanks to my friend Gospodin for documenting this one and sharing with me how he tested it,