VCSA Convergence: Failed to Get RPMs

One of my vSphere 6.7 U3 environments I am managing was still using external Platform Services Controller (PSC) from times when it was the prescribed architecture. That is no longer the case so to simplify my management I wanted to get rid of the external PSC via so called Convergence to embedded PSC.

Unfortunately although there is a very nice UI to do this it never worked for me. And I did try multiple times. The error I always ended up was:

Failed to get RPMs.

The /var/log/vmware/converge/converge.log log did not show any error, but what was peculiar there was this entry referring to download of VCSA 6.5.0 files?!

2019-10-29T16:02:01.223Z INFO converge currentURL = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/
2019-10-29T16:02:01.223Z INFO converge Manifest file = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/manifest/manifest-latest.xml

These are obviously not correct for my 6.7 U3 VCSA appliance. This VMware Communities thread finally pushed me in the right direction.

Here are the steps how to resolve this:

  1. Delete content of /root/vema directory on VCSA
  2. Download correct VCSA ISO installation media corresponding to the version of your VC. In my case it was the full 4 GB VMware-VCSA-all-6.7.0-14836122.iso. The patch media VMware-vCenter-Server-Appliance-6.7.0.41000-14836122-patch-FP.iso cca 2 GB large did not work.
  3. Mount the ISO to your VCSA
  4. Re-run the convergence via the UI

vCenter Server Issue: Recent Tasks Show xxx.label

I had an annoying issue in my lab. Some time ago when I performed vSphere 6.7 PSC convergence my vCenter would stop displaying proper names of tasks in the vSphere Clients UI (both Flex and H5) and show only their placeholders with names like xxx.label.

While there are some KB or communities articles about the issue (and fix) none of them was applicable to my situation (running vCenter Server 6.7U1). I thought that VCSA patches or even deploying new appliance with backup restore would fix it but it did not.

After a little research I found out that the issue is caused by missing catalog.zip file in the /etc/vmware-vpx/locale/ folder. I had another lab with the exactly same vCenter Server build deployed so I just copied that file and transferred it to my vCenter Server Appliance. After service restart via VAMI UI tasks names were back.

I do not know the root cause, but if you have the same issue, give it a go.

vCloud Availability: Replication of Powered-off VM

Just a short post about a feature I recently learned.

In vSphere Replication when you are configuring replication of powered-off VM you will get the following message:

The virtual machine is not powered on. Replication will start when the virtual machine is powered on.

The replication is actually configured and its placeholder VM is created in the recovery location (cloud) but the VM will stay in Not Active state.

Why is this? Immediate start of replication locks VM disks which means such VM would not be able to power-on until the initial sync is finished. But what if you want to replicate powered-off VMs for example templates that are never meant to run?

You can in fact force start the replication by right clicking the VM and selecting Sync Now, which asks confirmation question if we really want to do so as the VM will not be able the be powered on until the operation completes.

Is there a use case for this? As I mentioned this could be used for catalog sync as replication is much faster and efficient that OVF export / import.

vCloud Architecture Toolkit for Service Provider Update

The vCloud Architecture Toolkit for Service Provider website has been updated with new set of documents. All documents were re-branded with the new VMware Cloud Provider Program logos that replace the old vCloud Air Network brand.

My Architecting a VMware vCloud Director Solution for VMware Cloud Providers whitepaper has been refreshed to include vCloud Director 8.10 and 8.20 additions that were missing in the previous version. The current version of the document is 2.8 with August 2017 release date.

Here is summary of the new or updated topics:

  • Cell sizing
  • vCloud DB performance tips
  • New vCenter Chargeback Manager network metrics
  • vRealize Business for Cloud
  • vRealize Log Insight
  • vRealize Operations Manager
  • NSX Networking updates
  • Storage support
  • vCloud RBAC
  • Org VDC vSphere Resource Settings
  • VCDNI deprecation
  • New Org VDC Edge GW features
  • Distributed Firewall
  • VM Auto import
  • vCloud API for NSX
  • vCloud Director orchestrated upgrade

The document can be downloaded in PDF format or viewed online.

vCloud Director 8.20: VM Auto-import

While the biggest new feature of vCloud Director 8.20 is the access for tenants to NSX advanced services on Edge Gateways and distributed firewall, service providers will love automatic import of vCenter virtual machines.

Import of vCenter VMs under vCloud Director management was around for long time, but the VM had to be in powered off state. In vCloud Director 8.10, the possibility of running VM import was introduced (vCloud API only feature). However, vCloud Director 8.20 simplifies this even more:

The system administrator can just drag any vCenter VM into Org VDC resource pool and vCloud Director will automatically discover such VM and import it. The VM can be even in running state. This enables migration use cases, or allows service providers to easily offer self service access to their fully managed vSphere only environments by simply connecting them to vCloud Director.

In the picture below you can see Org VDC Resource Pool (ACME_PAYG) in vCenter hierarchy with two regular vCenter VMs that were dragged there. VM_C1 was running, while VM_P2 was powered-off.

resource-pool

The next picture shows automatically created vApps for these two imported VMs with the prefix Discovered.

vapps

While these vApps resemble regular vApps they are not real vCloud Director vApps until they are adopted. The adoption happens when the VM inside the vApp is somehow reconfigured.

By default VM discovery is enabled for every Organization in vCloud Director. It can be disabled in General Settings (UI or API)

general-settings

or with CMT command on VCD cell:

cell-management-tool manage-config -n managed-vapp.discovery.activated -v false

This behavior can be overridden at Org VDC level with API element <VmDiscoveryEnabled>.

Here are the differences between Discovered and Adopted vApp:

Discovered vApp

  • Looks like a regular vApp
  • Can have only one VM per discovered vApp
  • vApp contains API element <autoNature>true</autoNature>
  • When imported VM is deleted in VC or VCD, its vApp object will get automatically purged
  • Is owned by system
  • Is not subject to Org lease settings

Adopted vApp

  • Regular vCloud Director vApp
  • Can contain multiple VMs, vApp networks, etc.
  • Discovered vApp will get adopted when it is reconfigured (other than changing its name or description)

Other Considerations

  • Discovery process runs in the background every 3 mins
  • Failed VM import is retried after 60 mins. This can be changed with CMT command (example for 25 seconds):
    cell-management-tool manage-config -n managed-vapp.discovery.retry-delay-sec -v 25
  • The following VMs cannot be imported: Fault Tolerant VMs, VMs with creation/upload process in VC, templates, VCD shell VMs
  • VM must be connected to Org VDC network.
  • VM can be running or powered off.
  • VM does not need to use Org VDC storage policy. If it resides on unknown storage policy, it is automatically relocated to the default Org VDC storage policy during the adoption however the VM must be in powered off state.
  • VM with IDE controller must be in powered off state.
  • VM CPU/RAM resources are changed based on Org VDC allocation type.
  • VM resources are not subject to Org VDC allocation restrictions, but are charged against it.
  • VM name in VC remains intact until it is adopted and renamed
  • New vSphere 6.5 guest operating systems are not recognized and are imported as Other (32-bit) OS.

Update 4/24/2017

  • By default there is minimal VM age configured to 1 hour. It means VMs that were freshly reconfigured will be skipped for the import. This is to ‘settle’ the VMs first. The interval can however be changed with CMT command (age set to 60 seconds):
    cell-management-tool manage-config -n VM_DISCOVERY_MIN_AGE_SEC -v 60
  • Related to the minimal age, it is important that all cells and vCloud database are using the same time configuration. Not only the time must be correct but also the time zone must be configured identically.