{"id":446,"date":"2016-02-15T07:43:37","date_gmt":"2016-02-15T07:43:37","guid":{"rendered":"http:\/\/www.haxed.me.uk\/?p=446"},"modified":"2016-02-15T07:44:23","modified_gmt":"2016-02-15T07:44:23","slug":"removing-and-purging-all-rackspace-cloud-files-from-all-containers","status":"publish","type":"post","link":"https:\/\/haxed.me.uk\/index.php\/2016\/02\/15\/removing-and-purging-all-rackspace-cloud-files-from-all-containers\/","title":{"rendered":"Removing and purging all Rackspace Cloud Files from all containers"},"content":{"rendered":"<p>So, this one comes up a lot at work, and thanks to knowitnot.com I came accross this example using ruby and fog, its simple, dont runnaway!<\/p>\n<p>Regex lets you define the exact things you want to delete, but be careful. I&#8217;m using .* , I&#8217;ve removed it and replaced it with some_regex, to make sure people don&#8217;t accidentally nuke their cloud files \ud83d\ude00<\/p>\n<p>Install first <\/p>\n<pre>\r\nyum install rubygem-fog.noarch\r\n<\/pre>\n<pre>\r\n#!\/usr\/bin\/env ruby\r\n# Author: Jason Barnett <J@sonBarnett.com>\r\n\r\nrequire 'fog'\r\n\r\ndef delete_file(container, file_num, max_tries)\r\n  max_retries ||= max_tries\r\n  try = 0\r\n  puts \"(#{file_num} of #{container.files.count}) Removing #{container.files[file_num].key}\"\r\n  begin\r\n    container.files[file_num].destroy\r\n  rescue Excon::Errors::NotFound, Excon::Errors::Timeout, Fog::Storage::Rackspace::NotFound => e\r\n    if try == max_retries\r\n      $stderr.puts e.message\r\n    else\r\n      try += 1\r\n      puts \"Retry \\##{try}\"\r\n      retry\r\n    end\r\n  end\r\nend\r\n\r\ndef equal_div(first, last, num_of_groups)\r\n  total      = last - first\r\n  group_size = total \/ num_of_groups + 1\r\n\r\n  top    = first\r\n  bottom = top + group_size\r\n  blocks = 1.upto(num_of_groups).inject([]) do |result, x|\r\n    bottom = last if bottom > last\r\n    result << [ top, bottom ]\r\n\r\n    top    += group_size + 1\r\n    bottom =  top + group_size\r\n\r\n    result\r\n  end\r\n\r\n  blocks\r\nend\r\n\r\nservice = Fog::Storage.new({\r\n    :provider             => 'Rackspace',               # Rackspace Fog provider\r\n    :rackspace_username   => 'your_rackspace_username', # Your Rackspace Username\r\n    :rackspace_api_key    => 'your_api_key',            # Your Rackspace API key\r\n    :rackspace_region     => :ord,                      # Defaults to :dfw\r\n    :connection_options   => {},                        # Optional\r\n    :rackspace_servicenet => false                      # Optional, only use if you're the Rackspace Region Data Center\r\n})\r\n\r\ncontainers = service.directories.select do |s|\r\n  s.key =~ \/^some_regex\/  # Only delete containers that match the regexp\r\nend\r\n\r\nTOT_THREADS = 4\r\nthreads     = []\r\n\r\ncontainers.each do |container|\r\n  puts\r\n  puts \"-----------------------------------------\"\r\n  puts \"-- Removing _ALL_ objects from #{container.key}\"\r\n  puts \"-----------------------------------------\"\r\n  puts\r\n\r\n  #puts \"container.files.count: #{container.files.count}\"\r\n\r\n  ## separates the number of files into equal groups to distribute to each thread\r\n  mygroups = equal_div(0, container.files.count - 1, TOT_THREADS)\r\n\r\n  0.upto(TOT_THREADS - 1) do |thread|\r\n    threads << Thread.new([ container, mygroups[thread] ]) { |tObject|\r\n      tObject[1][0].upto(tObject[1][1]) do |x|\r\n        delete_file(tObject[0], x, 5)\r\n      end\r\n    }\r\n\r\n  end\r\n  threads.each { |aThread|  aThread.join }\r\n  puts \"Deleting #{container.key}\"\r\n  container.destroy\r\nend\r\n<\/pre>\n<p>The script works, supporting multiple threads:<\/p>\n<pre>\r\n(8178 of 10000) Removing conv\/2015\/04\/15\/08\/bdce78e4cab875e9e17eeeae051b1128.log.gz\r\n(3183 of 10000) Removing conv\/2015\/03\/17\/18\/0ea71bf7a834fc02b01c7f65c4eb23b0.log.gz\r\n(5685 of 10000) Removing conv\/2015\/04\/01\/08\/e4d6a7a2ee83d0be116cca6b1a92ad2a.log.gz\r\n(682 of 10000) Removing conv\/2015\/02\/26\/07\/163f33cf4e33c64139ab3bd7092a9478.log.gz\r\n(8179 of 10000) Removing conv\/2015\/04\/15\/09\/09ef6c4ecaa9341e76648b6e175db888.log.gz\r\n(5686 of 10000) Removing conv\/2015\/04\/01\/09\/23fa1dce145c8343efd8e0227fd41e35.log.gz\r\n(3184 of 10000) Removing conv\/2015\/03\/17\/18\/66cb5ac707c40777a1a78b1486e1c4f3.log.gz\r\n(683 of 10000) Removing conv\/2015\/02\/26\/07\/35f28368ed45b2fb7c7076c3b78eb008.log.gz\r\n(5687 of 10000) Removing conv\/2015\/04\/01\/09\/42882b705c2d7cfe5c726ddec3e457fc.log.gz\r\n(3185 of 10000) Removing conv\/2015\/03\/17\/18\/73dbb421db0093279138b6f69f246c06.log.gz\r\n(8180 of 10000) Removing conv\/2015\/04\/15\/09\/1a01e6b1779b76b5221b1c8c08398b00.log.gz\r\n(684 of 10000) Removing conv\/2015\/02\/26\/07\/44a59e91f6632383f18ff253e4404dba.log.gz\r\n(3186 of 10000) Removing conv\/2015\/03\/17\/18\/8f8ba5791ab0c1979b168517b229ef74.log.gz\r\n(5688 of 10000) Removing conv\/2015\/04\/01\/09\/8ff43c8a5de8a41ee9890686150c8c19.log.gz\r\n(8181 of 10000) Removing conv\/2015\/04\/15\/09\/28035bc7882fe36bb33da670e6c71ac0.log.gz\r\n(685 of 10000) Removing conv\/2015\/02\/26\/07\/7947d1b3f712d175281d33a6073531d9.log.gz\r\n(8182 of 10000) Removing conv\/2015\/04\/15\/09\/34a597377f63709a75c9fd6e8a68e073.log.gz\r\n(3187 of 10000) Removing conv\/2015\/03\/17\/18\/a92a58f166cc9bd1ff54a5f996bb776b.log.gz\r\n(5689 of 10000) Removing conv\/2015\/04\/01\/09\/9a728570208bcac32fd11f1581a57fbb.log.gz\r\n\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>So, this one comes up a lot at work, and thanks to knowitnot.com I came accross this example using ruby and fog, its simple, dont runnaway! Regex lets you define the exact things you want to delete, but be careful. &hellip; <a href=\"https:\/\/haxed.me.uk\/index.php\/2016\/02\/15\/removing-and-purging-all-rackspace-cloud-files-from-all-containers\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,15,14,9,7],"tags":[],"class_list":["post-446","post","type-post","status-publish","format-standard","hentry","category-api","category-cloud","category-cloud-files","category-linux","category-management-tools"],"_links":{"self":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/446","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/comments?post=446"}],"version-history":[{"count":2,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/446\/revisions"}],"predecessor-version":[{"id":448,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/446\/revisions\/448"}],"wp:attachment":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/media?parent=446"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/categories?post=446"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/tags?post=446"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}