Cannot use CBT: Soap fault – Reset CBT using PowerShell and VMware PowerCLI, the easy way

I was working with a customer who had several power outages, and we found our Veeam backups became highly unstable.  We kept receiving several failures pointing towards CBT as the culprit.

CBT is a VMware technology that Veeam allows you to leverage in a backup job to track changes, and backup only the changes that were made since the last Veeam backup.

Here’s the famous dreaded error that I’m sure you’ve seen, otherwise you wouldn’t be reading this post:

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/5167d01a-360063e4-c1b2-782bcb0d815f/VM-NAME/VM-NAME.vmdkDetail: ”, endpoint: ”

I did some research on the error above and landed on a VMware kb article.  This article described the steps necessary to reset CBT on each and every VM where Veeam would report a CBT fault.  The steps were basically as follows:

  1. Power off the VM
  2. Edit the advanced configuration parameters
  3. Manually change every instance of “ctkEnabled” to “false”
  4. Set the “scsi0:x.ctkEnabled” value to false for each disk of the VM in question
  5. Delete any -ctk.vmdk files found in the datastore
  6. Power on the VM
  7. Allow the VM to boot
  8. Power off the VM (resets the CBT table)
  9. Power on the VM again, for the 2nd time

I read these instructions and said to myself, “There has to be a better way to do this in a production environment.  Powering each and every virtual machine off and back on seems unnecessary.”

I also noticed that this CBT fix didn’t always work, as our Veeam jobs still came back with CBT errors.  Upon further investigation, I realized that the “false” values weren’t being changed back to “true” by VMware after I manually reset CBT on each VM.

I found a few excellent blogs, and came upon the below PowerShell script, but at first I had some trouble getting it to run properly.  Details are below the script which should clarify how to use this extremely powerful and time saving script.

In order to run this script, you’ll need to install VMware PowerCLI.  You can download it here: https://my.vmware.com/web/vmware/details?downloadGroup=PCLI550&productId=353

I found that adding the line below enabled me to connect to my vCenter server, so I’ve included that in the script.  Without this connection, the script won’t do much.

Connect-VIServer -Server vcenter-server.domain.local -Protocol https

The magic line at the end of the script is what resets CBT by taking a enabling CBT, taking a snapshot, and then immediately removing it, or “stunning” the VM with a snapshot.

Get-VM | where {$_.ExtensionData.config.ChangeTrackingEnabled} | Set-VMChangedBlockTracking -enabled:$true -applynow

The following steps are what I took to run this script and successfully reset CBT:

  1. Ensure VMware PowerCLI is installed
  2. Paste the entire script above into your PowerShell ISE environment
  3. Run the function “Load-PowerCLISnapin” starting on Line 1
  4. Modify line 15 to point to your vCenter server
  5. Execute line 15
    1. If you receive the error “Connect-VIServer : The term ‘Connect-VIServer’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.” – Manually run line 10: “Add-PSSnapin vmware.vimautomation.core”
  6. Place your Set-VMChangedBlockTracking statement on the very last line “Get-VM | where {$_.ExtensionData.config.ChangeTrackingEnabled} | Set-VMChangedBlockTracking -enabled:$true -applynow”
  7. Run the entire script

While the script is running, keep an eye on your vCenter activity.  You’ll notice the script automatically creating snapshots, and then deleting them.  Let it run its course, and retry your Veeam backup job.  You should no longer receive CBT errors!

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *