![\"screen-shot-2016-11-07-at-12-41-30-pm\"](\"http:\/\/www.haxed.me.uk\/wp-content\/uploads\/2016\/11\/Screen-Shot-2016-11-07-at-12.41.30-PM.png\")
<\/a><\/p>\nThis is a remarkably good question, because actually, not many tools are provided here, and the customer will, much like on many other CDN services, have to download those logs, and then process them. But that is actually not easier either, and I spent a good few weeks (albeit when I had time), trying to figure out the best way to do this. I dabbled with using tree to display the most commonly used logs, I played with piwik, awstats, and many others such as goaccess, all to no avail, and even used a sophisticated AWK script from our good friends in Operations. No luck, nothing, do not pass go, or collect $200. So, I was actually forced to write something to try and achieve this, from start to finish. There are 3 problems.<\/p>\n
1) how to easily obtain .CDN_ACCESS_LOGS from Rackspace Cloud Files to Cloud Server (or remote).
\n2) how to easily process these logs, in which format.
\n3) how to easily present these logs, using which application.<\/p>\n
The first challenge was actually retrieving the files.<\/strong><\/p>\n Naturally to perform this step above, you will need a working, and setup swiftly environment. If you don’t know what swiftly, is or understand how to set up a swiftly envrionment, please see this article I wrote on the subject of deleting all files with swiftly (The howto explains the environment setup first! Just don’t follow the article to the end, and continue from here, once you’ve setup and installed swiftly)<\/p>\n Fore more info see: Processing the Rackspace CDN Logs that we’ve downloaded, and organising them for further log processing <\/strong> The below script sits in the same folder as all of the containers<\/p>\n How to easily present these logs<\/strong><\/p>\n I kind of deceived you with the last step. Actually, because I have already done it, with the above script. Though, you will naturally need to have an httpd installed, and a documentroot in \/var\/www\/html, so make sure you install apache2:<\/p>\n De de de de de de da! da da!<\/p>\n Some little caveats:<\/p>\n Generating a master index.html file of all the sites <\/strong><\/p>\n I will expand the script to generate this automatically soon, but for now, leaving like this due to time constraints.<\/p>\n","protected":false},"excerpt":{"rendered":" So, this has been rather a bane on the life which is lived as Adam Bull. Basically, a large customer of ours had 50+ CDN’s, and literally hundreds of gigabytes of Log Files. They were all in Rackspace Cloud Files, … Continue reading \r\nswiftly --verbose --eventlet --concurrency=100 get .CDN_ACCESS_LOGS --all-objects -o .\/\r\n<\/pre>\n
\nhttps:\/\/community.rackspace.com\/products\/f\/25\/t\/7190<\/a><\/p>\n
\n\r\n# ls -al \r\ntotal 196\r\ndrwxrwxr-x 36 root root 4096 Nov 7 12:33 .\r\ndrwxr-xr-x 6 root root 4096 Nov 7 12:06 ..\r\n# used by my script\r\n-rw-rw-r-- 1 root root 1128 Nov 7 12:06 alldirs.txt\r\n\r\n# CDN Log File containers as we downloaded them from swiftly Rackspace Cloud Files (.CDN_ACCESS_LOGS)\r\ndrwxrwxr-x 3 root root 4096 Oct 19 11:22 dev.demo.video.cdn..com\r\ndrwxrwxr-x 3 root root 4096 Oct 19 11:22 europe.assets.lon.tv\r\ndrwxrwxr-x 5 root root 4096 Oct 19 11:22 files.lon.cdn.lon.com\r\ndrwxrwxr-x 3 root root 4096 Oct 19 11:23 files.blah.cdn..com\r\ndrwxrwxr-x 5 root root 4096 Oct 19 11:24 files.demo.cdn..com\r\ndrwxrwxr-x 3 root root 4096 Oct 19 11:25 files.invesco.cdn..com\r\ndrwxrwxr-x 3 root root 4096 Oct 19 11:25 files.test.cdn..com\r\n-rw-r--r-- 1 root root 561 Nov 7 12:02 generate-report.sh\r\n-rwxr-xr-x 1 root root 1414 Nov 7 12:15 logparser.sh\r\n\r\n# Used by my script\r\ndrwxr-xr-x 2 root root 4096 Nov 7 12:06 parsed\r\ndrwxr-xr-x 2 root root 4096 Nov 7 12:33 parsed-combined\r\n<\/pre>\n
\r\n#!\/bin\/bash\r\n\r\n# Author : Adam Bull\r\n# Title: Rackspace CDN Log Parser\r\n# Date: November 7th 2016\r\n\r\necho \"Deleting previous jobs\"\r\nrm -rf parsed;\r\nrm -rf parsed-combined\r\n\r\nls -ld *\/ | awk '{print $9}' | grep -v parsed > alldirs.txt\r\n\r\n\r\n# Create Location for Combined File Listing for CDN LOGS\r\nmkdir parsed\r\n\r\n# Create Location for combined CDN or ACCESS LOGS\r\nmkdir parsed-combined\r\n\r\n# This just builds a list of the CDN Access Logs\r\necho \"Building list of Downloaded .CDN_ACCESS_LOG Files\"\r\nsleep 3\r\nwhile read m; do\r\nfolder=$(echo \"$m\" | sed 's@\/@@g')\r\necho $folder\r\n echo \"$m\" | xargs -i find .\/{} -type f -print > \"parsed\/$folder.log\"\r\ndone < alldirs.txt\r\n\r\n# This part cats the files and uses xargs to produce all the Log oiutput, before cut processing and redirecting to parsed-combined\/$folder\r\necho \"Combining .CDN_ACCESS_LOG Files for bulk processing and converting into NCSA format\"\r\nsleep 3\r\nwhile read m; do\r\nfolder=$(echo \"$m\" | sed 's@\/@@g')\r\ncat \"parsed\/$folder.log\" | xargs -i zcat {} | cut -d' ' -f1-10 > \"parsed-combined\/$folder\"\r\ndone < alldirs.txt\r\n\r\n\r\n# This part processes the Log files with Goaccess, generating HTML reports\r\necho \"Generating Goaccess HTML Logs\"\r\nsleep 3\r\nwhile read m; do\r\nfolder=$(echo \"$m\" | sed 's@\/@@g')\r\ngoaccess -f \"parsed-combined\/$folder\" -a -o \"\/var\/www\/html\/$folder.html\"\r\ndone < alldirs.txt\r\n\r\n<\/pre>\n\r\nyum install httpd awstats\r\n<\/pre>\n
<\/a><\/p>\n
\n[root@cdn-log-parser-mother html]# pwd
\n\/var\/www\/html
\n[root@cdn-log-parser-mother html]# ls -al | awk '{print $9}' | xargs -i echo \" {} <\/a>
\" > index.html
\n<\/code><\/p>\n