{"id":191,"date":"2015-10-14T17:18:39","date_gmt":"2015-10-14T16:18:39","guid":{"rendered":"http:\/\/www.haxed.me.uk\/?p=191"},"modified":"2015-10-14T17:18:39","modified_gmt":"2015-10-14T16:18:39","slug":"restarting-a-xen-server-that-is-out-of-memory-thru-hypervisor","status":"publish","type":"post","link":"https:\/\/haxed.me.uk\/index.php\/2015\/10\/14\/restarting-a-xen-server-that-is-out-of-memory-thru-hypervisor\/","title":{"rendered":"Restarting a Xen Server that is out of memory thru Hypervisor"},"content":{"rendered":"<p>So, you have a xen server, but the virtual machine is not responding, what do you do? You login to the hypervisor and fix it, all right!<\/p>\n<p>Please note for sanitation that random strings have been used instead of real life UUID.<\/p>\n<p>Step 1: Connect Hypervisor<\/p>\n<pre>\r\nssh root@somehypervisoriporhostname\r\n<\/pre>\n<p>Step 2: Check Running Tasks (Task List)<\/p>\n<pre>\r\n[root@10-1-1-1 ~]# xe task-list\r\nuuid ( RO)                : ff9ca1a3-fc29-a245-1f28-2adc646114a2\r\n          name-label ( RO): Async.VM.clean_reboot\r\n    name-description ( RO):\r\n              status ( RO): pending\r\n            progress ( RO): 0.371\r\n\r\n\r\nuuid ( RO)                : aff56852-6db4-1ab3-b2b1-33e48c797dbb\r\n          name-label ( RO): Connection to VM console\r\n    name-description ( RO):\r\n              status ( RO): pending\r\n            progress ( RO): 0.000\r\n\r\n\r\n[root@10-1-1-1 ~]# xe task-list params=all\r\nuuid ( RO)                  : ff9ca1a3-fc29-a245-1f28-2adc646114a2\r\n            name-label ( RO): Async.VM.clean_reboot\r\n      name-description ( RO):\r\n            subtask_of ( RO): <not in database>\r\n              subtasks ( RO):\r\n           resident-on ( RO): 43b6096b-09cd-4890-b51b-56e50de573ff\r\n                status ( RO): pending\r\n              progress ( RO): 0.372\r\n                  type ( RO): <none\/>\r\n                result ( RO):\r\n               created ( RO): 20151014T15:01:17Z\r\n              finished ( RO): 19700101T00:00:00Z\r\n            error_info ( RO):\r\n    allowed_operations ( RO): Cancel\r\n\r\n\r\nuuid ( RO)                  : aff56852-6db4-1ab3-b2b1-33e48c797dbb\r\n            name-label ( RO): Connection to VM console\r\n      name-description ( RO):\r\n            subtask_of ( RO): <not in database>\r\n              subtasks ( RO):\r\n           resident-on ( RO): 43b6096b-09cd-4890-b51b-56e50de573ff\r\n                status ( RO): pending\r\n              progress ( RO): 0.000\r\n                  type ( RO): <none\/>\r\n                result ( RO):\r\n               created ( RO): 20151014T15:57:48Z\r\n              finished ( RO): 19700101T00:00:00Z\r\n            error_info ( RO):\r\n    allowed_operations ( RO):\r\n\r\n<\/pre>\n<p>I could see that there were two tasks running on this slice:<\/p>\n<pre>\r\n[root@10-1-1-1 ~]# xe vm-list name-label=slice10011111\r\nuuid ( RO)           : 4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff\r\n     name-label ( RW): slice10011111\r\n    power-state ( RO): running\r\n<\/pre>\n<p>This told me that the slice was running OK. So I am going to cancel the task pending for it<\/p>\n<pre>\r\n\r\n$ xe task-cancel uuid=ff9ca1a3-fc29-a245-1f28-2adc646114a2\r\n\r\n<\/pre>\n<p>Shutdown the server (HALT IT)<\/p>\n<pre>\r\n\r\n[root@10-1-1-1 ~]# xe vm-shutdown --force uuid=4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff\r\n[root@10-1-1-1 ~]# xe vm-list name-label=slice10011111\r\nuuid ( RO)           : 4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff\r\n     name-label ( RW): slice10011111\r\n    power-state ( RO): halted\r\n<\/pre>\n<p>Start the Virtual Machine<\/p>\n<pre>\r\n[root@10-1-1-1 ~]# xe vm-start uuid=4a9a5dfb-3c4a-b2bb-be7b-db3be6297fff\r\n<\/pre>\n<p>At the end I wanted to check if the instance was still causing a large swap as it was when it was running out of memory! That is the reason why I had to start the server.<\/p>\n<pre>\r\n(echo \"Slice IO_Read IO_Write Total\"; (for uuid in $(xe vbd-list params=uuid | awk '$5{print $5}'); do xe vbd-param-list uuid=$uuid | grep -P \"^\\s*(io_|vm-name-label|vdi-name-label|vdi-uuid|device)\" | awk '{if($1==\"vdi-uuid\") {hasswap=\"no\";vdi_uuid=$4;}}{if($1==\"vm-name-label\") name=$4; if($1==\"vdi-name-label\") {if ($4 ~ \/swap\/) {hasswap=\"yes\";name=name\"-swap\"}; if ($5 ~ \/ephemeral\/) name=name\"-eph\";} if($1==\"device\"){if($4==\"hda\" || $4==\"xvda\") name=name\"-root\"; if($4==\"xvdc\" && hasswap==\"no\") {vdicmd=\"xe vdi-list uuid=\"vdi_uuid\" params=name-description --minimal | grep swap >> \/dev\/null\"; swpname=system(vdicmd); if(swpname==0) name=name\"-swap\"};} if($1==\"io_read_kbs\") ioread=$4; if($1==\"io_write_kbs\") iowrite=$4}END{if(substr(name,0,9)!=\"XenServer\") print name\" \"ioread\" \"iowrite\" \"ioread+iowrite}'; done) | sort -k4n) | column -t\r\n\r\n<\/pre>\n<p>Job done!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, you have a xen server, but the virtual machine is not responding, what do you do? You login to the hypervisor and fix it, all right! Please note for sanitation that random strings have been used instead of real &hellip; <a href=\"https:\/\/haxed.me.uk\/index.php\/2015\/10\/14\/restarting-a-xen-server-that-is-out-of-memory-thru-hypervisor\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,11,5],"tags":[],"class_list":["post-191","post","type-post","status-publish","format-standard","hentry","category-cloud","category-openstack","category-xen"],"_links":{"self":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/191","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/comments?post=191"}],"version-history":[{"count":1,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/191\/revisions"}],"predecessor-version":[{"id":192,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/posts\/191\/revisions\/192"}],"wp:attachment":[{"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/media?parent=191"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/categories?post=191"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haxed.me.uk\/index.php\/wp-json\/wp\/v2\/tags?post=191"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}