Comments on: Random Reboots with VMware ESX 3.5 Update 3 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/ The weblog of an IT pro specializing in virtualization, storage, and servers Wed, 08 Feb 2012 17:13:47 +0000 hourly 1 http://wordpress.org/?v= By: slowe http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-43065 slowe Mon, 29 Dec 2008 13:45:42 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-43065 Kyle, I'll admit that VMware has had (in my opinion) too many problems with VMware HA, although I haven't seen many problems with VMware DRS (they are different features, after all, even though they are often used together). Keep in mind, though, that VMware HA was first introduced in VMware ESX 3.0, and I believe that many of the "problems" that were seen with VMware HA were really more a by-product of VMware failing to properly document the technology more than a defect of the technology itself (e.g., failing to educate users about the need for secondary Service Console connections, that sort of thing). Kyle, I’ll admit that VMware has had (in my opinion) too many problems with VMware HA, although I haven’t seen many problems with VMware DRS (they are different features, after all, even though they are often used together). Keep in mind, though, that VMware HA was first introduced in VMware ESX 3.0, and I believe that many of the “problems” that were seen with VMware HA were really more a by-product of VMware failing to properly document the technology more than a defect of the technology itself (e.g., failing to educate users about the need for secondary Service Console connections, that sort of thing).

]]> By: kyle http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-43059 kyle Sun, 28 Dec 2008 23:42:48 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-43059 Why is it that VMWARE has so many issues with HA and DRS. They sure do want to charge for it and ever since we got on board since v3.0 we have seen many many ha/drs issues. Why is it that VMWARE has so many issues with HA and DRS. They sure do want to charge for it and ever since we got on board since v3.0 we have seen many many ha/drs issues.

]]> By: Mike Karolow http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-43003 Mike Karolow Fri, 26 Dec 2008 17:03:28 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-43003 We've seen something similar on ESXi 3.5U3. For us, the guests stay up, but the host becomes completely unmanageable. This is apparently caused by the AAM service itself. VMware has told us it's an known bug, that it's been fixed, but they don't know when it will be released. Right now, they are targeting U4. Needless to say, we're a bit pissed that they don't want to release it as a standalone patch, since we're forced to disable HA. Here's the note from tech support: ...For your information, we have a new discovered an issue regarding HA on ESXi 3.5U3 and VC 2.5 U3, which causes the ESXi host to be unresponsive due to a large amount of error logs generated by AAM... ...If you want to confirm that this is the issue that you are facing, next time that you experience the issue, log into the Tech Support mode (or call us right way) and run the following command: #stat -f / File: "/" ID: 0 Namelen: 127 Type: visorfs Block size: 4096 Blocks: Total: 46792 Free: 15838 Available: 15838 Inodes: Total: 2048 Free: 950 If the line that indicates the number of free Inodes is reporting 0 free inodes, you have been affected by this issue. The issue does not affect all users and has been fixed, but the patch has not been released yet... We’ve seen something similar on ESXi 3.5U3. For us, the guests stay up, but the host becomes completely unmanageable. This is apparently caused by the AAM service itself.

VMware has told us it’s an known bug, that it’s been fixed, but they don’t know when it will be released. Right now, they are targeting U4. Needless to say, we’re a bit pissed that they don’t want to release it as a standalone patch, since we’re forced to disable HA.

Here’s the note from tech support:

…For your information, we have a new discovered an issue regarding HA on ESXi 3.5U3 and VC 2.5 U3, which causes the ESXi host to be unresponsive due to a large amount of error logs generated by AAM…

…If you want to confirm that this is the issue that you are facing, next time that you experience the issue, log into the Tech Support mode (or call us right way) and run the following command:

#stat -f /
File: “/”
ID: 0 Namelen: 127 Type: visorfs
Block size: 4096
Blocks: Total: 46792 Free: 15838 Available: 15838
Inodes: Total: 2048 Free: 950

If the line that indicates the number of free Inodes is reporting 0 free inodes, you have been affected by this issue. The issue does not affect all users and has been fixed, but the patch has not been released yet…

]]> By: Rick Vanover http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42939 Rick Vanover Wed, 24 Dec 2008 14:48:17 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42939 NiTRo: That isn't much of a fix by the way. NiTRo: That isn’t much of a fix by the way.

]]> By: slowe http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42936 slowe Wed, 24 Dec 2008 13:08:27 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42936 NiTRo, as I mentioned in the post, the reader has already disabled VM failure monitoring, so that can't be the fix for him. There is definitely something else going on in this situation. NiTRo, as I mentioned in the post, the reader has already disabled VM failure monitoring, so that can’t be the fix for him. There is definitely something else going on in this situation.

]]> By: NiTRo http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42932 NiTRo Wed, 24 Dec 2008 10:09:22 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42932 Hi Scott, Here is the fix : http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1007899&sliceId=1&docTypeID=DT_KB_1_1&dialogID=9786130&stateId=1%200%209784944 It works for me on about 20 ESX 3.5 Hi Scott,

Here is the fix : http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1007899&sliceId=1&docTypeID=DT_KB_1_1&dialogID=9786130&stateId=1%200%209784944

It works for me on about 20 ESX 3.5

]]> By: Iben Rodriguez http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42928 Iben Rodriguez Wed, 24 Dec 2008 06:06:00 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42928 Hi Scott, We have a client with this issue too. After updating to U3 some of the VMs were rebooting. We disabled the VM failure monitoring feature and so far so good. Many of our VMs (over 80) need to have their VMware Tools updated since the U3 update of the hosts so that may be contributing to the issue. We are monitoring the situation and do not have a root cause determined at this time. Good to know about the patch - we will investigate. Thanks! I b e n Hi Scott,

We have a client with this issue too. After updating to U3 some of the VMs were rebooting. We disabled the VM failure monitoring feature and so far so good. Many of our VMs (over 80) need to have their VMware Tools updated since the U3 update of the hosts so that may be contributing to the issue. We are monitoring the situation and do not have a root cause determined at this time.

Good to know about the patch – we will investigate.

Thanks!

I b e n

]]> By: Roger L http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42925 Roger L Wed, 24 Dec 2008 05:21:26 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42925 If I see this, i'll let you know. Thanks for the heads up. Roger L If I see this, i’ll let you know.

Thanks for the heads up.

Roger L

]]> By: Rick Scherer http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/comment-page-1/#comment-42922 Rick Scherer Wed, 24 Dec 2008 03:45:00 +0000 http://blog.scottlowe.org/2008/12/23/random-reboots-with-vmware-esx-35-update-3/#comment-42922 The VMotion/VMware Tools upgrade theory might be a possibility. Remember, when doing a VMotion the Virtual Machine is actually started on a new host, if VMware Tools is configured to automatically upgrade during boot it is quite possible a "hole" left open in U3 could turn this into an unwanted feature. VM Running -> Automatically Upgrade VMware Tools Enabled -> VMotion to New Host -> VMware Tools Upgrade Initiated -> Guest O/S Reboots This also makes me think about how in U3 of the Free Edition of ESXi the programmers left the Read/Write RCLI commands in there..... Whats going on over at the VMware QC department ? This is all speculation - I would be curious in knowing if it is the same VMs rebooting, or do they reboot once then never again. This would be more icing on the cake for the VMware Tools theory if it were the latter. The VMotion/VMware Tools upgrade theory might be a possibility. Remember, when doing a VMotion the Virtual Machine is actually started on a new host, if VMware Tools is configured to automatically upgrade during boot it is quite possible a “hole” left open in U3 could turn this into an unwanted feature.

VM Running -> Automatically Upgrade VMware Tools Enabled -> VMotion to New Host -> VMware Tools Upgrade Initiated -> Guest O/S Reboots

This also makes me think about how in U3 of the Free Edition of ESXi the programmers left the Read/Write RCLI commands in there….. Whats going on over at the VMware QC department ?

This is all speculation – I would be curious in knowing if it is the same VMs rebooting, or do they reboot once then never again. This would be more icing on the cake for the VMware Tools theory if it were the latter.

]]>