Esx Problem Hyperthreading Unmitigated

• Vmware Article 55636

I was always told from the first days building ESXi is that the first thing you do is disable hyperthreading. That is a legacy process back from the 2.x days, IIRC. The modern CPU scheduler has no problem with CPU placement with HT enabled. It is best to do your VM sizing based on full cores and let the CPU scheduler assign and use HT as needed. Error Warning ที่แสดงบน vCenter Server เกิดขึ้นหลังจาก Patch Update ESXi (CVE-2018-3646) XXX esx.

The L1 Terminal Fault (aka Foreshadow) bug is another speculative execution side channel attack that affects Intel Core processors and Intel Xeon processors only.

For VMware vSphere, there are some patches available as described in this document: VMSA-2018-0020. All patches have been released on August, 14th 2018.

*These patches DO NOT mitigate the Concurrent-context attack vector previously described by default. For details on the three-phase vSphere mitigation process please see KB55806and for the mitigation process for Workstation and Fusion please see KB57138.

**These patches include microcode updates required for mitigation of the Sequential-context attack vector. This microcode may also be obtained from your hardware OEM in the form of a BIOS or firmware update. Details on microcode that has been provided by Intel and packaged by VMware is enumerated in the patch KBs found in the Solution section of this document.

Of course, vSphere previous v5.5 are no more supported and there is no solution for them. And also vSphere 5.5 will go out of support soon!

But patching is just a step in the entire flow process:

As described in VMware KB 55806, the mitigation process for CVE-2018-3646 is divided into three phases:

1. Update Phase: Apply vSphere Updates and Patches

2. Planning Phase: Assess Your Environment

3. Scheduler-Enablement Phase: Enable the ESXi Side-Channel-Aware Scheduler

But of course, there is also the phase 0 needed to identify if your system is affected. In this RuneCast can help you (as happened with Spectre and Meltdown).

Then you have to apply the patches and in this case is very important to upgrade the vCenter FIRST and then the ESXi hosts. If you patching ESXi before their vCenter Server there can be a generic error such as xxx esx.problem.hyperthreading.unmitigated.formatOnHost not found xxx or esx.problem.hyperthreading.unmitigated.

For vCenter updating, with VCSA this could be managed simply with VAMI. For ESXi, VUM can help you.

Note that the L1TF related patches are (at least in vSphere 6.7) in the Non-Critical Host Patches baseline (sigh).

Note for VSAN users: seems that the VSAN baselines are not updated yet to include also this patch (in my case the VSAN 6.7 baseline remains on build 8169922). Maybe it’s just my problem, but I suggest to temporally add the non-critical patches baseline to proper patch your hosts.

After the host remediation each affected host will have a warning alarm with the proper reference at the issue:

A now? Now you have to fix your host by enabling the ESXi Side-Channel-Aware Scheduler:

Using the vSphere Web Client or vSphere Client:

1. Connect to the vCenter Server using either the vSphere Web or vSphere Client.
2. Select an ESXi host in the inventory.
3. Click the Manage (5.5/6.0) or Configure (6.5/6.7) tab.
4. Click the Settings sub-tab.
5. Under the System heading, click Advanced System Settings.
6. Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
7. Select the setting by name and click the Edit pencil icon.
8. Change the configuration option to true (default: false).
9. Click OK.
10. Reboot the ESXi host for the configuration change to go into effect.

1. Connect to the ESXi host by opening a web browser to https://HOSTNAME.
2. Click the Manage tab
3. Click the Advanced settings sub-tab
4. Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
5. Select the setting by name and click the Edit pencil icon
6. Change the configuration option to true (default: false)
7. Click Save.
8. Reboot the ESXi host for the configuration change to go into effect.

Using the CLI with the esxcli command:

1. SSH to an ESXi host or open a console where the remote ESXCLI is installed. For more information, see the http://www.vmware.com/support/developer/vcli/.
2. Check the current runtime value of the HTAware Mitigation Setting by running esxcli system settings kernel list -o hyperthreadingMitigation
3. To enable HT Aware Mitigation, run this command: esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
4. Reboot the ESXi host for the configuration change to go into effect.

Why is this remediation not active by default? Good question… the main answer seems to be the possible performance/capacity impact. How huge is performance degradation? Could be around 30% but a correct answer requires to read carefully the VMware KB 55767. The reality is that the degradation depends on the cluster usage level and is directly correlated to it.

But CPU usually is not a bottleneck and security by default must be the first approach… So, in my opinion, was better enable by default and then let the administrator choose if disable it.

Considering also that enable later means that you need a second host reboot, this becomes very time to consume, especially in a hyper-converged cluster where you need to reboot (usually) hosts sequentially one by one.

VMware has provided a tool (basically a PowerCLI script) to assist in performing both the Planning Phase and the Scheduler-Enablement Phase at scale. The tool is called HTAware Mitigation Tool and can be found in KB56931 along with detailed instructions on its usage, capabilities, and limitations.

Basically, you can extract the ZIP file and import the module. But the module is not signed, so the first step is to change the execution policy:

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted

Then, you can import the module:

Import-Module .HTAwareMitigation.psd1

Connect to the vCenter server:

Connect-VIServer -Server vcenter.lab.local

Start the analysis:

Get-HTAwareMitigationAnalysis

And remediate all the hosts (but first check for a possible performance impact):

Set-HTAwareMitigationConfig .output.csv -Enable

The remediation will only apply the advanced setting value, but you need to reboot your host after the process:

Processing ESXi host esxi1.lab.local ...
Unable to find HT-aware mitigation setting, ESXi host may not have been patched ...
Processing ESXi host esxi2.lab.local ...
Successfully updated 'VMkernel.Boot.hyperthreadingMitigation' to True

If a host has not been patched properly, the script will return a message like this: “ESXi host may not have been patched”.

Note that the script does not check if the setting is already present and also does not provide (yet) the reboot orchestration flow.

Important: Some vendors or other OSes suggest to disable Intel Hyperthreading in firmware/BIOS (you can also disable via ESXi using VMkernel.Boot.Hyperthreading). But VMware does not recommend this operation because it precludes potential vSphere scheduler enhancements and mitigations that will allow the use of both logical processors. As such, disablement of hyperthreading to mitigate the Concurrent-context attack vector will introduce unnecessary operational overhead as hyperthreading may need to be re-enabled in the future.

See also:

Related Posts

• New patches for VMware vCloud Suite

  VMware has recently released several patches for its vCloud Suite: VMware vCenter Server 5.1.0a and modules (Release Notes) VMware vSphere Data Protection 5.1.1 (Release Notes) vSphere Storage Appliance 5.1.1 (Release Notes) VMware vCloud Networking and Security 5.1.1 vCloud Director 5.1.1 Note also that View 5.1.x…

• vSphere 5 download

  Finally the vSphere 5 download is now available on VMware site: http://downloads.vmware.com/d/info/datacenter_cloud_infrastructure/vmware_vsphere/5_0 VMware ESXi 5.0 (Build 469512) VMware vCenter 5.0 (Build 456005) Also the official documents is now available: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

• vSphere 5 download

  Finalmente, dopo più di un mese dal Lancio ufficiale di VMware vSphere 5, è disponibile la versione per il download, direttamente dal sito di VMware: http://downloads.vmware.com/d/info/datacenter_cloud_infrastructure/vmware_vsphere/5_0 VMware ESXi 5.0 (Build 469512) VMware vCenter 5.0 (Build 456005) Oltre al download dei binari, sono disponibili anche i…

Andrea Mauro

Virtualization, Cloud and Storage Architect. Tech Field delegate.VMUG IT Co-Founder and board member. VMware VMTN Moderator and vExpert 2010-20 and vExpert Pro. Dell TechCenter Rockstar 2014-15. Microsoft MVP 2014-16. Veeam Vanguard 2015-19. Nutanix NTC 2014-20.Several certifications including: VCDX-DCV, VCP-DCV/DT/Cloud, VCAP-DCA/DCD/CIA/CID/DTA/DTD, MCSA, MCSE, MCITP, CCA, NPP.

An area of confusion that seems to arise quite often is that around that of the Hyperthreading (HT) feature within ESX Virtual Infrastructure Client (VIC) being displayed as being ‘Inactive’.

Here is a link to Wikipedia that outlines what Hyperthreading is about and how it works. This is another article with easy to follow information on the different CPU architectures.

If you see Hyperthreading within the VIC set to ‘Inactive’ you’re probably thinking “Hey, what’s going on here? I’ve got a modern processor that can do this sorta stuff.”

Well, the answer is.. it depends.

If you have a single core processor and it is a Pentium 4 or early’ish (vague technical term 🙂 ) Xeon then if it likely that you have Hyperthreading on the CPU’s die. Sometimes it must be enabled in the BIOS first. This article tells you what to look for. Also, this list from Intel outlines which Intel processors and chipsets support Hyperthreading technology.

Hyperthreading disappeared from the Intel range of CPU’s with the introduction of the Merom, Conroe, and Woodcrest range of multi-core processors.

So if you are using a semi-recent PC/Server to run ESX then chances are you don’t have Hyperthreading technology on your CPU(s) and hence it is showing up as ‘Inactive’ within the Virtual Infrastructure Client.

Vmware Article 55636