VMWare ESXi and Software iSCSI Targets

During a recent upgrade of VMWare ESXi from 4.1 to 5.5, I encountered a problem: after the upgrade, systems stored on a shared iSCSI storage platform did not work. In fact, a number of them were damaged. Files could be read and downloaded from the datastore with no issues, but writes to the datastore failed.

A number of symptoms appeared, most tellingly in the VMWare client console about connectivity to the datastore being lost and then restored. However, the vmkernel logs on the hosts themselves showed more interesting failures, specifically reservations and the failure of SCSI ‘WRITE_SAME’ 0x93 errors. This led me to a post about the Linux SCST target software, stating that this doesn’t support this command and to disable VAAI/acceleration features in VMWare.

This did solve the problem and it appears that VMWare has been a bit dumb here. The options are marked as ‘requires compliant hardware’ inside VMWare but 1) they are turned on by default, and 2) are applied to targets with a HW acceleration value of ‘Unknown’, rather than being restricted to validated hardware. Poor design decision in my opinion.

To turn off these features, you need to edit the following on each host:

ESXi Configuration > Advanced Settings

/VMFS3/HardwareAcceleratedLocking > Set to 0

/DataMover/HardwareAcceleratedMove > Set to 0

/DataMover/HardwareAcceleratedInit > Set to 0