We can all agree that larger vSphere environments bring many challenges. One of them is updating all of the VMware Tools on the numerous VM guests. One of the solutions our team decided to use was to enable the VMware Tools Update during power cycling, previously discussed here. This worked for us well in vSphere 4.x and updated the majority of Tools in our environment. VMware has changed this feature in vSphere 5 however.
Our environments were recently upgraded to vSphere 5 and everything was looking great! Then we had our routine maintenance weekend. This would update well over half of the tools and we would get the rest on the next maintenance weekend. That was the plan.
Throughout the next week, we started receiving reports of users logging into their servers and being prompted about a VMware Tools install and the server needed to be rebooted for the changes to take effect. The user naturally clicked ‘No’ but the server proceeded to reboot. Wait? The user clicked ‘No”, right? That’s what they said. The first couple we didn’t think much of it. Then some of our team experienced this issue. So we knew this just wasn’t a “user error.”
We got VMware on the phone to investigate. They had no reports of this happening anywhere. After a long discussion, we were informed the order that the VMware Tools upgrade during power cycling was changed in vSphere 5. Instead of updating the tools on the way down, it now updates the tools after the server has already rebooted. I don’t know why this was changed but this now requires an additional reboot on top of the reboot that initiated the tools install.
Wait, what about the users clicking ‘No’ when they were asked to reboot or not and it still reboots. As time went on and more info was gathered, we found this was only happening on Windows 2003 x86 servers. The user was presented with the following dialog box:
The instances we saw, the above dialog box was always shown. Servers that weren’t Windows 2003 x86 showed a second dialog box that was specific to VMware Tools and accepted the No as an option. Windows 2003 x86 servers however, did not present this second dialog box and proceeded to reboot. The Windows event logs then had the following event:
VMware did some further troubleshooting and had us perform further testing. When we encountered these dialog boxes in a Windows 2003 x86 server within the vSphere Console, the server did not reboot. It did reboot when a user had logged in using a remote desktop client.
VMware still has not given us a definitive answer as to why this is occurring as this is the first they have seen of it. I will post an update here when we get a verdict. Also, ensure the new order of Tools upgrade during power cycle will fit your needs.
—UPDATE—
VMware never confirmed our testing that was described above. However, we are sticking to our findings and give caution to your updates.