Nexus 1000V VEM Upgrade Issue

I was upgrading Nexus 1000V in my lab to the latest version 4.2(1)SV2(2.1). First you need to upgrade Nexus VSM (Virtual Supervisor Module) by uploading kickstart and system files to it and then installing them. This was done without any issues. The next step is to upgrade the VEM modules on every host connected to the Nexus 1000V switch.

I did not use Nexus in a while so I was surprised how Cisco changed the upgrade process. It has following steps:

1. Network admin prepares the Nexus switch for upgrade by upgrading the VSM and notifying vSphere admin that the switch is ready to be upgraded by running following command on the VSM:
   vmware vem upgrade notify
2. The vSphere admin sees the notification in the switch summary tab in the Networking inventory view and must click Apply Upgrade. He basically confirms that VMware vSphere Update Manager (VUM) is available and configured properly. (Btw you see the message incorrectly says vSphere Distributed Switch instead more generic Distributed vSwitch.)
   
3. The message above changes to “An Upgrade … is in progress” and network admin can proceed with the upgrade.
   
4. This is done by running following command on the VSM:
   vmware vem upgrade proceed
5. Update Manager now tries to upgrade the VEMs on each host. You can observe the issued vCenter tasks (Enter maintenance mode, Check, Install, Exit Maintenance Mode)
6. Network admin can monitor the status on VSM by running:
   show vmware vem upgrade status
7. If all hosts are upgraded he will see:
   Nexus1000V-02# show vmware vem upgrade status
   
   Upgrade VIBs: System VEM Image
   Upgrade Status: Upgrade Complete in vCenter
   Upgrade Notification Sent Time: Sun Sep 15 19:05:18 2013
   Upgrade Status Time(vCenter): Sun Sep 15 19:06:42 2013
   Upgrade Start Time: Sun Sep 15 19:08:06 2013
   Upgrade End Time(vCenter): Sun Sep 15 19:09:14 2013
   Upgrade Error:
   Upgrade Bundle ID:
   VSM: VEM500-201306160100-BG
   DVS: VEM500-201306160100-BGand can finalize the upgrade by issuing

   vmware vem upgrade

   which will get rid of the upgrade in progress message from step #3.

However on my system step #5 was failing with an error: “The host returns esxupdate error code:99. An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.”

Examining the esxupdate log files I could see that there was some problem with the new VEM vib file:

esxupdate: esxupdate: ERROR: ValueError: Cannot merge VIBs Cisco_bootbank_cisco-vem-v160-esx_4.2.1.2.2.1.0-3.1.1, Cisco_bootbank_cisco-vem-v160-esx_4.2.1.2.2.1.0-3.1.1 with unequal payloads attributes: ([<vmware.esximage.Vib.Payload object at 0x88374ec>], [<vmware.esximage.Vib.Payload object at 0x884632c>])

The problem was caused by conflict of the same VIB metadata obtained from the default CIsco repository (https://hostupdate.vmware.com/software/VUM/PRODUCTION/csco-main/csco-depot-index.xml) and directly from VSM. Unfortunately I do not know a way to remove a patch already in VUM database (other than hacking the database) so instead I did following:

1. Disconnect VUM from the internet (so it cannot download Cisco patch metadata when it starts).
2. Stop VUM service
3. Purge VUM database (this will get rid of all patches, baselines, etc.). See KB 2043170
4. Start VUM service and uncheck the custom Cisco download source in the VUM Download Settings configuration.
   
5. Connect VUM to the internet again.

The installation will now proceeded without problems.

Note: As the actual VIB binary is downloaded by VUM from the VSM and not the internet so make sure that VUM has connectivity to VSM.