{"id":695,"date":"2016-07-06T18:18:36","date_gmt":"2016-07-06T17:18:36","guid":{"rendered":"http:\/\/www.haxed.me.uk\/?p=695"},"modified":"2016-07-06T18:18:36","modified_gmt":"2016-07-06T17:18:36","slug":"checking-file-integrity-cloud-files-post-upload-file","status":"publish","type":"post","link":"https:\/\/haxed.me.uk\/index.php\/2016\/07\/06\/checking-file-integrity-cloud-files-post-upload-file\/","title":{"rendered":"Checking File integrity with Cloud Files, post upload file"},"content":{"rendered":"

So, as you may already be aware, I am working on a lightweight backup script called obscene redundancy’. An redundant backup software capable of 18 replicas of data to Rackspace Cloud Files API service. It’s so redundant… it’s obscene redundancy.<\/p>\n

<\/span><\/em><\/strong>For more details visit the project URL:
\nhttps:\/\/github.com\/aziouk\/obsceneredundancy\/<\/a><\/span><\/p>\n

Today, I was discussing with my colleague, that it was all very well uploading your tar to cloud files, but, wouldn’t you really like to know if the file you uploaded is completely identical number of bits, and order? Enter, Cloud Files ‘HEAD’and Etag. Our MD5 friend.<\/p>\n

What I did to improve the obscene redundancy script was quite simple here:<\/p>\n

\r\n# We define a variable that takes the 'Etag' (MD5Sum) value for the cloud files archive\r\ncfmd5sum=$(swiftly --conf swiftly-configs\/swiftly-${SHORT_REGION,,}.conf head\r\n\"${BACKUP_DEST}\/${FILE}\" | grep -i Etag | awk '{print $2}')\r\n\r\n# We Define a variable that generates an 'MD5Sum' for the local file archive\r\nlocalmd5sum=$(md5sum \"$BACKUP_DIR\"\/\"$FILE\")\r\n\r\necho \"Checking Data integrity of Cloud Files upload to $REGION\"\r\necho \"Cloud Files Archive MD5:\u00a0 $cfmd5sum\u00a0 ....... Local File Archive MD5: $localmd5sum\"\r\n\r\n# If these values\r\nif [[ \"$cfmd5sum\" -ne \"$localmd5sum\" ]];\r\nthen\r\necho \"VALUES NOT EQUAL\"\r\necho \"$REGION CRC OK...\"\r\nelse\r\necho \"VALEUS EQUAL\r\necho \"$REGION CRC missing, in error, or NOT OK...\"\r\nfi\r\n\r\n<\/pre>\n
After all this I found that the script wasn’t working properly… so I did some debugging about this to check, at least, first of all , the length of each variable. <\/p>\n
\r\n   if [[ \"$cfmd5sum\" == \"$localmd5sum\" ]]; then\r\n                        echo \"VALUES EQUAL, (local md5sum length given first)\"\r\n                        echo \"$localmd5sum\"| wc -L\r\n                        echo \"$cfmd5sum\"| wc -L\r\n\r\n\r\n                        echo \"$REGION CRC OK...\"\r\n                else\r\n                        echo \"VALUES NOT EQUAL\"\r\n                        echo \"$localmd5sum\"|wc -L\r\n                        echo \"$cfmd5sum\"|wc -L\r\n                        echo \"$REGION CRC missing, in error, or NOT OK...\"\r\n                fi\r\n<\/pre>\n
The output shown me that the variable length was different. At this stage I’ve no idea why, but will add updates here. I’m going to commit this to obsceneredundancy because proof of concept is working and valid, as shown by the output of the script. (i.e. the method is fine, it’s just the way the string is compared in the if, statement, I suspect it is to do with special character or \\n characters as I had before. So, when I made this addition to the multi-dc-backup.sh script.. the output now looks like:<\/p>\n
\r\nCreating Container in LON for obsceneredundancy\r\n\r\nLON: Backing up ...\r\nSource: \/var\/www\/ ---> Dest: cloudfiles:\/\/LON\/obsceneredundancy\/varwww-2016-07-06-6bd657e9-d268-4883-9f40-3859f690aadb.tar.gz\r\n\r\nChecking Data integrity of Cloud Files upload to BACKUP_TO_LON\r\nCloud Files Archive MD5:  65147eb66f8bbeff03a229570b0a1be7  ....... Local File Archive MD5: 65147eb66f8bbeff03a229570b0a1be7  \/var\/backup\/varwww-2016-07-06-6bd657e9-d268-4883-9f40-3859f690aadb.tar.gz\r\nVALUES NOT EQUAL\r\n107\r\n32\r\nBACKUP_TO_LON CRC missing, in error, or NOT OK...\r\nlon: COMPLETED OK 15504796\/15504796\r\nORD: Not backing up ...\r\n\r\n\r\n\r\nCreating Container in IAD for obsceneredundancy\r\n\r\nIAD: Backing up ...\r\nSource: \/var\/www\/ ---> Dest: cloudfiles:\/\/IAD\/obsceneredundancy\/varwww-2016-07-06-6bd657e9-d268-4883-9f40-3859f690aadb.tar.gz\r\n\r\nChecking Data integrity of Cloud Files upload to BACKUP_TO_IAD\r\nCloud Files Archive MD5:  65147eb66f8bbeff03a229570b0a1be7  ....... Local File Archive MD5: 65147eb66f8bbeff03a229570b0a1be7  \/var\/backup\/varwww-2016-07-06-6bd657e9-d268-4883-9f40-3859f690aadb.tar.gz\r\nVALUES NOT EQUAL\r\n107\r\n32\r\nBACKUP_TO_IAD CRC missing, in error, or NOT OK...\r\niad: COMPLETED OK 15504796\/15504796\r\nDFW: Not backing up ...\r\n<\/pre>\n
As we can see the 107 (localmd5size) and the 32 (cloudfilesmd5size) are different! I’ve no idea why, since when echoing the variables they look the same. I suspect gremlins and Trolls. A fresh head tomorrow will probably solve this in a few minutes!<\/p>\n
Cheers &
\nBest wishes,
\nAdam<\/p>\n","protected":false},"excerpt":{"rendered":"
So, as you may already be aware, I am working on a lightweight backup script called obscene redundancy’. An redundant backup software capable of 18 replicas of data to Rackspace Cloud Files API service. It’s so redundant… it’s obscene redundancy. … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19,15,39,14,37,10,48,7,57,11,12,47,20],"tags":[],"class_list":["post-695","post","type-post","status-publish","format-standard","hentry","category-bash","category-cloud","category-cloud-backup","category-cloud-files","category-disaster-recovery","category-filesystem","category-git","category-management-tools","category-obsceneredundancy","category-openstack","category-openstack-api","category-swiftly","category-webhosting"],"_links":{"self":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/695","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/comments?post=695"}],"version-history":[{"count":1,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/695\/revisions"}],"predecessor-version":[{"id":696,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/695\/revisions\/696"}],"wp:attachment":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/media?parent=695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/categories?post=695"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/tags?post=695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}