Hello from Veeam Support!
Recently, a customer contacted us with the very curious case. The SureBackup job was failing with the generic error message during snapshot creation phase: “The specified feature is not supported by this version.” There were several VMs included in the SureBackup job, but only one specific VM failed. Moreover, the source backup job itself worked perfectly, including the affected VM.
Isolating the issue
As this is a generic error message, it could have several reasons to appear. For example, the Free ESXi version has some limitations and therefore is not supported. However, we were sure that the version of ESXi was not Free, because other VMs were processing just fine in a SureBackup job, as well as the affected VM in a regular backup job.
As we have already stated before, that issue persisted only for one specific VM, so the first step was to find what was so special about this exact machine. At the very first glance, the main difference for this machine (let’s call it VM01) was its size — it was significantly bigger in size than others with one of its disks more than 2TB.
Keeping that fact in mind, we referred to the Veeam Backup & Replication task log file:
[timestamp] <56> Info [VimApi] Create snapshot, ref "vm-12345", name "VEEAM_SUREBACKUP_SNAPSHOT", description "", memory "False", quiesce "False"
[timestamp] <56> Error An error occurred while taking a snapshot: The specified feature is not supported by this version.
[timestamp] <56> Error CreateSnapshot failed, vmRef vm-12345, timeout 1800000, snName VEEAM_SUREBACKUP_SNAPSHOT, snDescription , memory False, quiesce False (System.Exception)
[18.09.2017 18:17:18] <56> Error at Veeam.Backup.ViSoap.CSoapConnection.CreateSnapshot(String vmRef, Int32 timeout, String name, String description, Boolean memory, Boolean quiesce)
[timestamp] <56> Error at Veeam.Backup.SureBackup.Operations.CSnapshotVmEngine.InternalProcess()
[timestamp] <56> Error at Veeam.Backup.SureBackup.Operations.CSnapshotVmEngine.Process()
[timestamp] <56> Error The operation is not supported on the object. (Veeam.Backup.ViSoap.ViServiceFaultException)
[timestamp] <56> Error VimApi.NotSupported
[timestamp] <56> Error at Veeam.Backup.ViSoap.CSoapService.ExecuteAndWaitForCompletion(IServiceOperationAsync operation, Nullable`1 timeout)
[timestamp] <56> Error at Veeam.Backup.ViSoap.CSoapConnection.CreateSnapshot(String vmRef, Int32 timeout, String name, String description, Boolean memory, Boolean quiesce)
Honestly speaking, it wasn’t really helpful at that moment but one thing was clear — according to the error stack, the message was generated by VMware vSphere API.
Always check VMware logs
That is why we decided to check the VMware log files and figure out the possible reason for such behavior. To make our search easier, we just used the error string as a search query and here we go:
timestamp info hostd[XXXX] [XXXX sub=DiskLib opID=XXXXX-XX-XXXXX user=vpxuser:DOMAIN\NAME] DISKLIB-LIB_CLONE : Could not get default Object Type for seSparse - The specified feature is not supported by this version:24.
timestamp info hostd[XXXXXX] [XXXX sub=Libs opID=XXXXX-XX-XXXXX user=vpxuser:DOMAIN\NAME] SNAPSHOT: SnapshotBranchDisk: Failed to branch disk: '/vmfs/volumes/XXXXXX-XXXX/XX_XXXX/vmdk_name.vmdk' -> '/vmfs/volumes/vSAN:XXXXXX-XXXXXXX/SureBackup/XXXXX-XXXXX-XXXX-XXXXX-XXXXX/vmdk_name-000001.vmdk' : The specified feature is not supported by this version (24)
The first thing that draws our attention is that the error message contained a specific disk name. We checked and that was the large disk (> 2TB) VMDK we saw before.
We also saw a mention of seSparse disk type (space-efficient sparse format), which reminded us that VMware vSphere is using seSparse format instead of VMFS-deltas (vmfssparse) when making a snapshot for disks larger than 2TB (see VMware KB 2058287).
From this moment, we knew the issue was somehow related to the seSparse redo logs disk type, but it didn’t explain what was exactly unsupported until we remembered one important detail — the original VM and SureBackup virtual lab were located on vSAN datastore. It’s a well-known fact that SureBackup is using Veeam vPower NFS store to raise VMs from it, but redo logs for SureBackup VMs are redirected to the production datastore (which is vSAN in our case):
[timestamp] <56> Info [SureBackup] [VM01] [ReconfigVm] [Line 104] workingDir = "/vmfs/volumes/vSAN:XXXXXX-XXXXXXX/SureBackup/XXXXX-XXXXX-XXXX-XXXXX-XXXXX"
[timestamp] <56> Info [SureBackup] [VM01] [ReconfigVm] [Line 105] snapshot.redoNotWithParent = "TRUE"
Basically, we had a heterogeneous VMs storage configuration with basic disks located on NFS datastore and redo logs located on vSAN. As this was working flawlessly for all the machines other than VM01, we concluded that there was something wrong between seSparse and vSAN compatibility.
Fast searching lead us to VMware vSAN documentation with the following statement: “Virtual SAN does not support SE Sparse disks.”
You may notice that documentation is related to vSphere v5.5, and in our case the vSphere version was 6.0. We contacted VMware and confirmed that the limitation still exists, even for 6.0 and 6.5.
At this point, the puzzle pieces fit into one big picture — you couldn’t have a parent disk on NFS/VMFS storage and a seSparse redo log on vSAN as snapshot inherits the VMFS_ type as per VMware documentation.
You could ask why are backups working fine on vSAN for large (> 2TB) disks if seSparse is not supported by vSAN? That’s simple – the VMware hypervisor is using VSANSPARSE redo logs type while creating a snapshot for machines with a parent disk located on vSAN.
Useful notes to keep in mind:
- VSANSPARSE are created on vSAN disks
- VMFS_sparse or seSparse are created on regular VMFS disks depending on disk size or VMFS version (snapshots on VMFS6 will be seSparse regardless of size)
- During redirect on another datastore, the snapshot disk type is inherited from the parent disk type.
Solution
We suggested the virtual lab migrate to a non-vSAN datastore. Therefore, the type of snapshot remains the same during redirection.
The same issue can be encountered when having VMs on vSAN and running instant recovery for VMs with large disks (> 2TB). And the same solution is valid for the instant recovery snapshot redirection.
Conclusion
Based on our research, it turned out that the above scenario covers more than instant recovery and SureBackup issues — in some cases, even hot-add mode will not work. You need to attentively check the types of datastores and snapshots that are used during the operation so you can avoid unsupported setup.
