28678 Hyper-V checkpoints are not deleted

Summary

Sometimes, Hyper-V VMMS might encounter a problem where it fails to automatically remove and merge backup checkpoints for certain virtual machines. This can result in the persistence of the backup Hyper-V checkpoint, which can be verified in Hyper-V Manager, and subsequent backup attempts may fail. To resolve this issue, you can follow the steps below.

Cause

When VSS is executed on the drive where the VM of Hyper-v host is stored, the status of VM becomes [Backing up…].
Because the checkpoint cannot be created in this status, the HyperBack task fails with the error below.

07/07/2020 01:13:11.027 NJHyperVGetRCTOneDiskPathAndTrackingID return error 0X7014 Level=ERROR [183]
07/07/2020 01:13:11.028 {IDS_STRING3018}Exit Code: 28678
07/07/2020 01:13:11.028 [28678] Unknown AIP error

Workaround

1. Check and removal of relevant checkpoints

1. Run Windows PowerShell as an administrator on the affected Hyper-V host.
2. You can first check all associated Hyper-V backup checkpoints for each virtual machine with the following command. Use the following command specifying the Hyper-V host and VM:
   Get-VMSnapshot -ComputerName "HOST" -VMName "VM"
   Note: The fully qualified domain name (FQDN) of your Hyper-V host may need to be used.
3. To remove the Hyper-V backup checkpoints, you need to run the following command:
   Get-VMSnapshot -ComputerName "HOST" -VMName "VM" | Remove-VMSnapshot
4. This process can take some time. Make sure you have at least 5% free space on the hard disk.

2. Clear Hyper-V cache

If a Hyper-V backup checkpoint is still displayed in Hyper-V Manager after using the PowerShell solution, but the virtual machine is already accessing the normal virtual hard disks * .VHD or * .VHDX, the Hyper-V snapshot cache is still there remained.

• If your virtual machine no longer has any * .AVHD or *.AVHDX files, the display in the Hyper-V Manager is only the cache of the Hyper-V snapshot. To clear the cache, proceed as follows:
  1. Shut down the affected virtual machine.
  2. Stop the Hyper-V VMMS "Hyper-V Virtual Machine Management" service.
  3. Start the Hyper-V VMMS service "Hyper-V Virtual Machine Management".
  4. Start the virtual machine.

 

3. Resolving permission issues when deleting Hyper-V backup checkpoints

You receive the error event 19100 in the Windows Event Viewer on the Hyper-V host:

Hyper-V-VMMS 19100 'VM1' Background Disk Merge Could Not Complete: General Access Denied Error (0x80070005)

The cause of this problem is related to incorrect permissions. To resolve this issue, additional permissions must be granted to the folder, virtual disk files, and checkpoint files, and to the VMMS process. First determine the GUIDs of the virtual guests who are affected by the problem.

1. Start PowerShell as administrator and enter the following command:
   get-vm | fl name, id
   
   Example output:
   
   Name : VM1
   Id : d3599536-222a-4d6e-bb10-a6019c3f2b9b
   
   Name : VM2
   Id : a0af7903-94b4-4a2c-b3b3-16050d5f80f
    

   For each GUID of a virtual machine, set full control for the VMMS process with the following command:

   icacls "YOUR PATH" /grant "NT VIRTUAL MACHINE\YOUR ID:(OI)F"
   
   

   Example:

   icacls "C:\ClusterStorage\Volume1\VM1" /grant "NT VIRTUAL MACHINE\d3599536-222a-4d6e-bb10-a6019c3f2b9b:(OI)F"

   icacls "C:\ClusterStorage\Volume1\VM2" /grant "NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2:(OI)F"

2. Repeat the process if required for other virtual guests.

Target Product

– ActiveImage Protector 2018Update or later