There are varying opinions within the greater VMware community about upgrading VM hardware versions. Newer virtual machine hardware versions introduce new features, new guest OS support, better compatibility and performance with CPU vulnerability mitigations, better support for modern CPU features, better security defaults, and so on.
Upgrading virtual machine hardware changes the virtual hardware presented to the guest operating system, just as if you placed a boot device from a physical server into a newer physical server. These changes can vary in risk, may require more than one reboot, and may require human interaction to complete. This forms the basis for many of the opinions that recommend leaving VM hardware versions alone, including the opinion espoused in the relevant Broadcom KB article. However, the KB’s opinion is also likely colored by the desire to avoid fielding support cases about obscure Microsoft Windows hardware plug & play issues. I get it. You worry about getting caught between two vendors pointing fingers at each other with this sort of issue. However, with a little bit of testing, it isn’t a big deal.
You can mitigate the risk of the VM hardware version upgrade in four ways:
- First, and most helpful, is the fact that a virtual machine snapshot also captures virtual hardware configuration, too. That means that if you snapshot a VM before you upgrade, you can revert the snapshot and the VM hardware version will revert, too. This makes testing easy.
- Second, keep your VMware Tools up to date. The VMware Tools include drivers for virtual hardware, so having those drivers already installed is important. Also keep in mind that VMware drivers are available via Windows Update (as well as in the Windows installation media) so if you use WSUS to manage updates you should include those.
- Third, keep your virtual machines free of unnecessary virtual hardware. This is covered in the VMware Security Configuration Guides as a way to reduce attack surface, but it also de-complicates your VMs. Less is actually more sometimes. You can usually safely remove the CD/DVD, the SATA controller it is attached to, and any USB/XHCI controllers.
- Last, schedule the hardware compatibility upgrade to only take place after a graceful guest OS shutdown. This step ensures that a virtual hardware change does not catch you off guard during a vSphere HA restart of a VM, a power outage, or another complicating incident.
In practice I’ve found that a virtual machine running Windows may simply need a second reboot once it boots again. Not a huge deal on a one-off basis, but at scale that can be more problematic. Testing will indicate whether that is an issue or not, though.
Should I Upgrade My Virtual Machine Hardware Version?
Should you upgrade? If the time is right, yes. One good time to do this is when you are building a new template VM. You should always use the latest version for a new guest OS template VM. Another good time is after a major VMware vSphere or VMware Cloud Foundation upgrade, so you can take advantage of new features.
If you are in a mixed environment you may not want to upgrade all the way to the latest version, or you might want to wait. For example, if you run vSphere 7, which supports up to hardware version 19, and vSphere 8, which supports hardware version 21. If you need to move virtual machines between the environments you may want to wait to do this work until everything is at vSphere 8.
Recommendations for Upgrading VM Hardware Compatibility
Run the latest version you are able, ideally the latest version available in the major vSphere version you run. Use VM Hardware 13 (vmx-13) or newer. Version 13 introduces important performance and security improvements for CPU vulnerability mitigations.
Do not upgrade virtual appliances from vendors. VMware, in particular, does not support upgrading the virtual hardware version of product appliances like the VCSA. However, mistakes happen (it’s easy to accidentally select all your VMs and forget to deselect vCenter Server… just sayin’), and while it seems to work just fine, your mileage may vary.
Take snapshots of virtual machines prior to upgrading, but do not forget to remove the snapshot later. Snapshots are a lot less troublesome on vSAN ESA nowadays, though.
When scheduling virtual hardware compatibility upgrades use the “Only upgrade after normal guest OS shutdown” to help ensure that a compatibility update does not complicate an unplanned incident or HA event.
Upgrade VMware Tools prior to the compatibility upgrade, so that you have the latest drivers and functionality.
Test! This is a particularly easy upgrade to test, as a snapshot will revert everything. Testing will let you gain familiarity with the process and get a sense of whether it will work or not, and what any “gotchas” might be (like the need for a second guest OS reboot).
Don’t do this work manually. Instead, work it out to schedule the hardware compatibility upgrade for most of your fleet of VMs. Thousands and thousands of organizations have done this, and you can, too.
Schedule staged rollouts. Once your testing is complete, choose a handful of VMs to go first. Take snapshots of them. Let them automatically update when they do their monthly patching. Observe them. When it works, schedule a bunch more. When that batch works, schedule twice as many again. If you run into problems assess if they’re just “one-off” issues or something more systemic.
Good luck!
Posted in “Virtualization” and “VMware” — you might be interested in other posts in those categories.
