Another question we had today was about tracking traffic on a Linux Server, i.e. daily, hourly, weekly and monthly statistics on bandwidth usage. This helps us keep the machine secure, and notice any untoward or extremely high traffic; another easy install:

apt-get install vnstat
# or
yum install vnstat

 

vnstat -u -i eth0

So, a customer wanted to restrict the amount of bandwidth used by his cloud server, with a kind of self-QOS like service.

To do this is actually quite simple, wondershaper is the magic wand for you.

# Install
apt-get install wondershaper 
# or
yum install wondershaper

#Configure for interface
sudo wondershaper eth1 256 128  

(Values are in kbit, so be sure to remember!)

If this isn’t suitable for you and you want something to restrict user space, like applications, then use trickle.

$ trickle -u {up} -d {down} {program}  

i.e. for restricting firefox to 8Kb/s up and down use:

    $ trickle -u 8 -d 8 firefox  

After many people popularly requesting to me to document how I configured my Raspberry Pi 2 with retropie to play Arcade Games. Many different consoles are supported. It really is quite extensive. From what I understand, the Operating System is ‘retroarch’ with some modifications, and emulation station, a kind of ‘suite’ of Console and Arcade Emulators of various different kinds.

    3do (lr-4do) (EXPERIMENTAL)
    Amiga (UAE4ALL2), (UAE4ARM)
    Amstrad CPC (CPC4Rpi), (lr-cap32)
    Apple II (Linapple)
    Atari 2600 (Stella), (lr-stella)
    Atari 5200, and Atari 8 bit series: 400, 800, 1200XL, 600XL, 800XL, 130XE, XEGS (Atari800)
    Atari 7800 (lr-prosystem)
    Atari Jaguar (lr-virtualjaguar) (EXPERIMENTAL)
    Atari Lynx (lr-handy)
    Atari ST/STE/TT/Falcon (Hatari)
    CoCo (XRoar)
    Commodore 64 (Vice)
    Dragon 32 (XRoar)
    Dreamcast (Reicast) (EXPERIMENTAL!)
    FinalBurn Alpha (PiFBA), (lr-fba), (lr-fba-next)
    Genesis/Megadrive (DGEN), (lr-Genesis-Plus-GX), (lr-picodrive)
    Game Gear (Osmose), (lr-Genesis-Plus-GX)
    Game Boy (lr-gambatte)
    Game Boy Color (lr-gambatte)
    Game Boy Advance (gpSP), (lr-gpSP), (lr-vba-next), (lr-mgba)
    Intellivision (jzIntv)
    Macintosh (BasiliskII)
    MAME (AdvanceMAME), (MAME4ALL-Pi), (lr-imame4all), (lr-mame2003), (lr-mame2010)
    MasterSystem (lr-Genesis-Plus-GX), (lr-picodrive), (Osmose)
    MSX (lr-fmsx), (lr-bluemsx)
    Neo Geo (GnGeo-Pi), (PiFBA), (lr-fba), (lr-fba-next)
    Neo Geo Pocket (Color) (lr-mednafen-ngp)
    Nintendo 64 (Mupen64plus), (lr-mupen64plus)
    Nintendo DS (lr-desmume) (EXPERIMENTAL)
    Nintendo Entertainment System (lr-fceumm), (lr-nestopia)
    PC (DOSBox), (rpix86)
    PC Engine/TurboGrafx-16 (lr-mednafen-pce-fast)
    Ports
        Cave Story (lr-nxengine)
        Descent 1 & 2 (DXX-Rebirth) (EXPERIMENTAL)
        DOOM (lr-prboom), (ZDOOM)
        Duke Nukem 3D (EDuke32)
        KODI (EXPERIMENTAL)
        Minecraft Pi Edition (EXPERIMENTAL)
        OpenTTD (openttd) (EXPERIMENTAL)
        OpenTyrian (EXPERIMENTAL)
        Quake Series (lr-tyrquake), (ioQuake3)
        Super Mario War
        SuperTux
        Wolfenstein 3D
    PlayStation 1 (lr-pcsx-rearmed), (pcsx-rearmed)
    PSP (lr-ppsspp), (ppsspp) (EXPERIMENTAL)
    ScummVM
    Sega 32X (lr-picodrive)
    Sega CD (lr-picodrive)
    Sega Saturn (lr-yabause) (EXPERIMENTAL!)
    Sega SG-1000 (lr-Genesis-Plus-GX)
    Super Nintendo Entertainment System (PiSNES), (snes9x-rpi), (lr-armsnes), (lr-catsfc), (lr-pocketsnes), (lr-snes9x-next)
    Vectrex (lr-vecx)
    Videopac or Odyssey2 (lr-o2em)
    Virtual Boy (lr-beetle-vb) (EXPERIMENTAL)
    WonderSwan (Color) (lr-mednafen-wswan)
    Zmachine (Frotz)
    ZX Spectrum (FBZX), (Fuse), (lr-fuse)

Now down to business, actually getting this up and running. Pre-requisites are Rasperry Pi 2 and some power, and a controller or keyboard, that is about it. Oh of course, you need a super microsd card!

Step 1. Download the Retropie Images to your hard disk.

RPI 3.2.1:
http://downloads.petrockblock.com/images/retropie-v3.2.1-rpi2.img.gz

Step 2. Download Win32Disk Imager (this is used to put the image you downloaded above onto the microsd)

http://sourceforge.net/projects/win32diskimager/

Step 3. Open up win32disk imager

It's easy. Make it look a bit like the image below and follow these steps:

Find the retropie-v3.2.1.-rpi2.img.gz file and unzip with winrar, or winzip, or something like 7zip, pkzip, or any uncompressing app should do.
Select that as the 'source image'.
Select the destination (whichever device which is the sdcard in your sdcard reader) , usually something like D:\ , E:\ or F:\, for me it's H:\.

Step 4. Take that fresh microsd card and pop it into the rasperry pi 2. Connect USB power and keyboard, preferably.

mmm. Tastes good. Like Raspberries. This is pretty much it. Very easy. But what about the ROMS?! arghh??? Yeah, true. The next two steps for that, but first, it makes sense to expand the filesystem. It basically lets you use the full space on the SDCARD, try not to think about it. It’s important , though.

You can press F4 on the keyboard at any time to drop to command prompt and you can type raspi-config yourself to expand the filesystem. But here is to do it from the ‘retropie’ menu selection in emulation station, the GUI that automatically starts with Retropie.

Step 5. Get a generic USB stick, 16GB or 32GB would be ideal, especially if you have a lot of ROMS. Preferably one with an LED on it which shows when it is read/writing. This is handy and you’ll see why later.

Plop the USB stick in your computer, format with FAT32 by right clicking device in windows and selecting ‘FORMAT’.
Create a folder in the highest directory for the USB stick, call the new folder ‘retropie’,
H:\retropie

Plop the USB stick into the retropie, and be patient, wait for the USB stick to flash, and then stop flashing if you have one that flickers LED when its being read/written to

Once the LED on USB stick stops flickering, remove it from the Rasperry pi2, and put it back into your computer. Copy the roms into the relevant folders, i.e. for n64, the n64 folder, for nintendo entertainment system the ‘nes’ folder, for megacd the ‘segacd’ folder, and for genesis the genesis/megadrive folder. You get the idea, it’s really simple. Once you’ve finished copying them to USB stick, take the USB stick out of your pc again.

Place the USB stick in the retropie, it has a service that checks for any changes on the USB stick, and it automatically rsyncs (synchronises the files on the USB stick with it’s sd card. If you were awesome and used an USB stick that flickers LED when writing/reading, wait for it to stop flickering before removing from rpi!

Now restart the thing (and this might not even be necessary), and then the emulation station will show the icons for the devices you’ve added roms for (don’t panic if you can’t see some emulators, if you didn’t put roms in the folder, then it won’t show up, think about it , this makes sense for almost 100 different emulators shipped, not to see the ones you have no roms for! It got me for a little while until I put that together.

Some really super duper important caveats:

Segacd requires copyrighted BIOS uploaded to the /opt/rpi/BIOS folder, or some such. Will document this more later.

Thanks to my colleague Aaron, he deserves the credit for this, this nice little one-liner for determining largest directories usage on a Linux Server.

Handy for customers having issues with running out of space and helping them identify unnecessary files, runaway logs, and other disk fillers.

FS='/';resize;clear;date;df -h $FS; echo "Largest Directories:"; du -hcx --max-depth=2 $FS 2>/dev/null | grep [0-9]G | sort -grk 1 | head -15 ;echo "Largest Files:"; nice -n 19 find $FS -mount -type f -print0 2>/dev/null| xargs -0 du -k | sort -rnk1| head -n20 |awk -F'\t' '{printf "%8d MB\t%s\n",($1/1024),$NF}'

From my colleague Jan.

DOM=$(xe vm-list name-label=instance-2fgfgf763fgfgidsfiusdf783hj params=dom-id –minimal)
UUID=$(uuidgen)

# Verify Variables set correctly
echo Domain: $DOM UUID: $UUID

# Perform resetnetwork
xenstore-write /local/domain/$DOM/data/host/$UUID ‘{“name”:”resetnetwork”,”value”:””}’

# wait few seconds
xenstore-read /local/domain/$DOM/data/guest/$UUID

# if that is problematic then check if request is hanging:
xenstore-ls /local/domain/$DOM/data

Download Package

wget http://boot.rackspace.com/files/nova-agent/nova-agent-Linux-x86_64-1.39.0.tar.gz

Steps

http://bootrackspacecom.readthedocs.org/en/latest/nova_agent/

This one doesn’t really come up that often at work, but it was some harmless fun I had this morning, when I thought, ‘is it possible to take some cert and key files’ and then build json around it with echo >> and sed the privateKey and publicCertificate into their rightful places in an lb.json file, and then curl a request against the Rackspace Load Balancer API.

So whats the point/joy of doing this? Well, it allows you to add certificate mappings with relative ease. Just plop your .cert and your .key file in the certificates folder, and the script can do all the rest. Of course you need to provide your username and APIKEY, but you always need to do that when making requests to the API. It’s also worth noting the TOKEN is generated automatically.

Next I will write a script that generates self signed certificates and then injects them in, so literally no user action is required. Obviously this isn’t going to be that useful, but if I connected it to an API-like certificate making service that was authorised ssl reseller, it would be a pretty tight product, I would go so far to say awesome.

Here is how I achieved it:

#!/bin/bash

USERNAME='mycloudusernamehere'
APIKEY='apikeyhere'

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`


echo '
{
  "certificateMapping": {
     "hostName": "my.com",
     "certificate": "' > lb.json

cat certificates/private.key | sed ':a;N;$!ba;s/\n/\\n/g' > certificates/private.short
cat certificates/public.cert | sed ':a;N;$!ba;s/\n/\\n/g' >  certificates/public.short

cat certificates/public.short >> lb.json
echo '", "privateKey": "' >> lb.json
cat certificates/private.short >> lb.json
echo '" } }' >> lb.json


curl -v -H "X-Auth-Token: $TOKEN" -d @lb.json -X POST -H "content-type: application/json"  https://lon.loadbalancers.api.rackspacecloud.com/v1.0/10011111/loadbalancers/157089/ssltermination/certificatemappings

My colleague referred this as a ‘sneaky way’ to parse Json. He is indeed correct, I am quite sneaky, but if it’s simple and it works, then booyah. This is what the lb.json file looks like after it’s created by the above shellscript.

{
  "certificateMapping": {
     "hostName": "my.com",
     "certificate": "
-----BEGIN CERTIFICATE-----\nMIIC/TCCAeWgAwIBAgIJAP5bHAHitdeoMA0GCSqGSIb3DQEBBQUAMBUxEzARBgNV\nBAMMCnd3dy5teS5jb20wHhcNMTUxMjAyMDkzNjEzWhcNMjUxMTI5MDkzNjEzWjAV\nMRMwEQYDVQQDDAp3d3cubXkuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEAxcSqtsqQUrFEY327avnR7uxxO6svkvPzzv7ANUhZ142OYZ4727sgDJeA\nbKllpxrCqZfnVDfd+YcloLukcHoEKYC0/6R/nygZbaXwA0WGLhNX+L43MEsldtGx\ntk3eO0Gs3B1t9na9NEjTO0YMxXsgnXrTZFUB2bD/UL8TkdtoWdlVgPwtIPeVyGZF\nhj3dBzO6SPvfixTrZLz8EAZ95I1bOHR+0UnXHZ6z7Bh+fKD4NQbXTSEFH/0HoAXV\nfHm5BxwsheFrQm3/0fisraArPFhDVfOrkCcVta8MniJn6SMtk8Us66ACIdl7uydM\nHqLqs29TQOGyB9nIxTL1h4T7+tbHiwIDAQABo1AwTjAdBgNVHQ4EFgQUOpK+W3FR\nUcttjZtmCEYwlXUon3AwHwYDVR0jBBgwFoAUOpK+W3FRUcttjZtmCEYwlXUon3Aw\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAL8Oo1nrykXCr2hYBg6on\nXLi5Tehsp6495U8xZygUL0fK08TUovjnVjln3qEsarotREZaTtmAjVrNZwYJrrn7\nHoxoOiccHw0FL3UfPR4q2oS+Z94Q+ZG9kXptO84nPV6WAx96lOXfPCVast9CsaVs\nkZRyZBQtYO+Mh53zxhouqNG69/OvSdDz4tCGi6MTZWmZGhnGx7SaTMITfOeK7IU8\nN4sMZwmHHsubKVZvcB0xN8Q+1Zwv7SPUuOi+OSd7v7llxlJ4bu2UQ55cLWb697dZ\nNCAChW2xsi157XUfPGnayfO/DbEQFdRULkKStY8o2jiu7GaovWtPVHY0kxjQKfY4\nQg==\n-----END CERTIFICATE-----\n
", "privateKey": "
-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAxcSqtsqQUrFEY327avnR7uxxO6svkvPzzv7ANUhZ142OYZ47\n27sgDJeAbKllpxrCqZfnVDfd+YcloLukcHoEKYC0/6R/nygZbaXwA0WGLhNX+L43\nMEsldtGxtk3eO0Gs3B1t9na9NEjTO0YMxXsgnXrTZFUB2bD/UL8TkdtoWdlVgPwt\nIPeVyGZFhj3dBzO6SPvfixTrZLz8EAZ95I1bOHR+0UnXHZ6z7Bh+fKD4NQbXTSEF\nH/0HoAXVfHm5BxwsheFrQm3/0fisraArPFhDVfOrkCcVta8MniJn6SMtk8Us66AC\nIdl7uydMHqLqs29TQOGyB9nIxTL1h4T7+tbHiwIDAQABAoIBAQCj+HBWR9KrTSBX\noQqAIoslnlIv17oFDFDMAbnZM5iRuGMhmrEkeJyU9BPdhAGtL+nP9Qsub3eSiLPw\n9ULcor3Kr1TiVEAf9H5Iw/kgrUcX8p/Qs91MJDH2ttuyPBOSa9xnT9s5Kq+qpurD\nzUuPfIvJJeoY2MZE+JRnHVWbbB+zxZ9dCzXGFsx5u4Yq1dI85vxB+5pzvPDJtQwy\nsIGszREHm6m1qeCXB3Hh3gU5un8fLh4kMfKAGcJEgS9AHXsKDgPSHOsCO3LnHGTW\nVyMtDpMEqq3rs/C2p533IDJylq+eoelnMnl8s2ieyxNjRCZLClQjpZdFgdULyPEK\nhWPOZgXBAoGBAP35DDvmWunIjEZxIlKnLn+vtz6kX+99HWpNouM3XegGp7rF8/7t\nlbwmYr8G290CjZNEjtvKW5vIPTkE8ZK8hZsmdbWkf92GUo1/cbIrZcfqBkC38rck\n5bWqXtyzzguRVMFj2UhqfYto4w6/bsA/8phnI5G0i8Op/VqE9rN5wpthAoGBAMdY\nxim7Clb54d1lCkq+uz3FA3WQkCEiq9ou6okEV3RqkqxqVjJW7Bjh0q4GSW8u2Xvh\nVaGx4Jk8Q9LCTB3x70TRTfAbg3RZqetclDPRan0tg1WHVcjzEqeS5xVa7uCBnBut\naTiT37MBzZRAh8oZQLOuFX+Y/pC5UTgv/p+glZZrAoGAOz23m9VMyZGNHvVO00bJ\n8uDS9pqzAhMGJIC9iRCmJ/Q9dbStCH702XF+wR5hdLkeuwZX6G7YVYsstLsxek/d\nPmaHOHqJlOu7H+RlafDzieFN2hTOWegSaQC3pfWPD2W0BnQ6/8hPRpCNvifrNo70\nEJamVltt6pMhVNcFELJLMaECgYBphjC//mbmy7gofkgIcRalCBlgrnndUIEwKg21\nIjs5QQELi+69Dw5Dzaa8wE83L9GopguyYHrIIwK0Gm44m81Q3IspQyc+/Afas1Mw\nava39NPE/rMGgMWrNzRkNZKl/XYpoI5GiOCt3ZJ5m/9FmECL3Oc8eDypV7AK0j0z\nOsp0qQKBgEhaQnwVN8+el/GEW/+weESP1GHWdvtDedeE19DOXnTNpR+V/wOpcpC7\n4oOlWARVCj4gGE+ugBSeX4slQmzu1L6p0npQ8jEIfbxR1znn+RK4EWKQKsoyfb1u\nw4ewR/Bwubv6iL7ct0FLFSjJXeNMc1+VmVpBTICpV0PrKbCP9uTw\n-----END RSA PRIVATE KEY-----
" } }

This is what I can use to quickly get terminal access to a console of a server.
Faster than trying to reach the control panel, or look in backend.

supernova lon get-vnc-console uuidgoeshere novnc

So, we occasionally get customers who are having issues creating a new server from an image of a previous server. Normally this is caused by the nova-agent not being set to start on boot, or the xe-linux-distribution being missing from the VM. It’s possible to check whether a virtual machine is configured right and I put together this little piece with the help of my colleague and friend Zoltan.

1. Checking if Nova Agent is installed and can be started

# /etc/init.d/nova-agent start

2. Check if nova-agent and xe-linux-distribution is running on VM

# ps auxf | grep nova
# ps auxf | grep xe-daemon

If processes called nova-agent or xe-daemon return then you know they are running OK.

3. Ensure that both services do start during boot

# chkconfig nova-agent on
# chkconfig xe-linux-distribution on

For Debian and Ubuntu Systems you may need to use

update-rc.d -f nova-agent defaults

Once you confirm that these services are running it’s safe to take an image, and create a new VM with it. These 2 processes need to be running because when the new server is built the way that the VM gets its networking set is using xenstore and novaagent to retrieve and set the network interfaces file with correct ip, subnet and gateway.

– A