Nexus 1000V VEM Upgrade Issue

I was upgrading Nexus 1000V in my lab to the latest version 4.2(1)SV2(2.1). First you need to upgrade Nexus VSM (Virtual Supervisor Module) by uploading kickstart and system files to it and then installing them. This was done without any issues. The next step is to upgrade the VEM modules on every host connected to the Nexus 1000V switch.

I did not use Nexus in a while so I was surprised how Cisco changed the upgrade process. It has following steps:

  1. Network admin prepares the Nexus switch for upgrade by upgrading the VSM and notifying vSphere admin that the switch is ready to be upgraded by running following command on the VSM:
    vmware vem upgrade notify
  2. The vSphere admin sees the notification in the switch summary tab in the Networking inventory view and must click Apply Upgrade. He basically confirms that VMware vSphere Update Manager (VUM) is available and configured properly. (Btw you see the message incorrectly says vSphere Distributed Switch instead more generic Distributed vSwitch.)
    Are you ready for upgrade?
  3. The message above changes to “An Upgrade … is in progress” and network admin can proceed with the upgrade.
    Upgrade in process
  4. This is done by running following command on the VSM:
    vmware vem upgrade proceed
  5. Update Manager now tries to upgrade the VEMs on each host. You can observe the issued vCenter tasks (Enter maintenance mode, Check, Install, Exit Maintenance Mode)
  6. Network admin can monitor the status on VSM by running:
    show vmware vem upgrade status
  7. If all hosts are upgraded he will see:
    Nexus1000V-02# show vmware vem upgrade status

    Upgrade VIBs: System VEM Image

    Upgrade Status: Upgrade Complete in vCenter
    Upgrade Notification Sent Time: Sun Sep 15 19:05:18 2013
    Upgrade Status Time(vCenter): Sun Sep 15 19:06:42 2013
    Upgrade Start Time: Sun Sep 15 19:08:06 2013
    Upgrade End Time(vCenter): Sun Sep 15 19:09:14 2013
    Upgrade Error:
    Upgrade Bundle ID:
    VSM: VEM500-201306160100-BG
    DVS: VEM500-201306160100-BGand can finalize the upgrade by issuing

    vmware vem upgrade

    which will get rid of the upgrade in progress message from step #3.

However on my system step #5 was failing with an error: “The host returns esxupdate error code:99. An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.”

Examining the esxupdate log files I could see that there was some problem with the new VEM vib file:

esxupdate: esxupdate: ERROR: ValueError: Cannot merge VIBs Cisco_bootbank_cisco-vem-v160-esx_4., Cisco_bootbank_cisco-vem-v160-esx_4. with unequal payloads attributes: ([<vmware.esximage.Vib.Payload object at 0x88374ec>], [<vmware.esximage.Vib.Payload object at 0x884632c>])

The problem was caused by conflict of the same VIB metadata obtained from the default CIsco repository ( and directly from VSM. Unfortunately I do not know a way to remove a patch already in VUM database (other than hacking the database) so instead I did following:

  1. Disconnect VUM from the internet (so it cannot download Cisco patch metadata when it starts).
  2. Stop VUM service
  3. Purge VUM database (this will get rid of all patches, baselines, etc.). See KB 2043170
  4. Start VUM service and uncheck the custom Cisco download source in the VUM Download Settings configuration.
    VUM Download Settings
  5. Connect VUM to the internet again.

The installation will now proceeded without problems.

Note: As the actual VIB binary is downloaded by VUM from the VSM and not the internet so make sure that VUM has connectivity to VSM.

One thought on “Nexus 1000V VEM Upgrade Issue

  1. Hi,

    We hit the same problem, but we are not able to solve the problem, since we cannot get the right VEM version. Any suggestion?

    BR, Marko

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.