vCenter Server Issue: Recent Tasks Show xxx.label

I had an annoying issue in my lab. Some time ago when I performed vSphere 6.7 PSC convergence my vCenter would stop displaying proper names of tasks in the vSphere Clients UI (both Flex and H5) and show only their placeholders with names like xxx.label.

While there are some KB or communities articles about the issue (and fix) none of them was applicable to my situation (running vCenter Server 6.7U1). I thought that VCSA patches or even deploying new appliance with backup restore would fix it but it did not.

After a little research I found out that the issue is caused by missing catalog.zip file in the /etc/vmware-vpx/locale/ folder. I had another lab with the exactly same vCenter Server build deployed so I just copied that file and transferred it to my vCenter Server Appliance. After service restart via VAMI UI tasks names were back.

I do not know the root cause, but if you have the same issue, give it a go.

Advertisements

vCloud Availability: Replication of Powered-off VM

Just a short post about a feature I recently learned.

In vSphere Replication when you are configuring replication of powered-off VM you will get the following message:

The virtual machine is not powered on. Replication will start when the virtual machine is powered on.

The replication is actually configured and its placeholder VM is created in the recovery location (cloud) but the VM will stay in Not Active state.

Why is this? Immediate start of replication locks VM disks which means such VM would not be able to power-on until the initial sync is finished. But what if you want to replicate powered-off VMs for example templates that are never meant to run?

You can in fact force start the replication by right clicking the VM and selecting Sync Now, which asks confirmation question if we really want to do so as the VM will not be able the be powered on until the operation completes.

Is there a use case for this? As I mentioned this could be used for catalog sync as replication is much faster and efficient that OVF export / import.

vCloud Architecture Toolkit for Service Provider Update

The vCloud Architecture Toolkit for Service Provider website has been updated with new set of documents. All documents were re-branded with the new VMware Cloud Provider Program logos that replace the old vCloud Air Network brand.

My Architecting a VMware vCloud Director Solution for VMware Cloud Providers whitepaper has been refreshed to include vCloud Director 8.10 and 8.20 additions that were missing in the previous version. The current version of the document is 2.8 with August 2017 release date.

Here is summary of the new or updated topics:

  • Cell sizing
  • vCloud DB performance tips
  • New vCenter Chargeback Manager network metrics
  • vRealize Business for Cloud
  • vRealize Log Insight
  • vRealize Operations Manager
  • NSX Networking updates
  • Storage support
  • vCloud RBAC
  • Org VDC vSphere Resource Settings
  • VCDNI deprecation
  • New Org VDC Edge GW features
  • Distributed Firewall
  • VM Auto import
  • vCloud API for NSX
  • vCloud Director orchestrated upgrade

The document can be downloaded in PDF format or viewed online.

vCloud Director 8.20: VM Auto-import

While the biggest new feature of vCloud Director 8.20 is the access for tenants to NSX advanced services on Edge Gateways and distributed firewall, service providers will love automatic import of vCenter virtual machines.

Import of vCenter VMs under vCloud Director management was around for long time, but the VM had to be in powered off state. In vCloud Director 8.10, the possibility of running VM import was introduced (vCloud API only feature). However, vCloud Director 8.20 simplifies this even more:

The system administrator can just drag any vCenter VM into Org VDC resource pool and vCloud Director will automatically discover such VM and import it. The VM can be even in running state. This enables migration use cases, or allows service providers to easily offer self service access to their fully managed vSphere only environments by simply connecting them to vCloud Director.

In the picture below you can see Org VDC Resource Pool (ACME_PAYG) in vCenter hierarchy with two regular vCenter VMs that were dragged there. VM_C1 was running, while VM_P2 was powered-off.

resource-pool

The next picture shows automatically created vApps for these two imported VMs with the prefix Discovered.

vapps

While these vApps resemble regular vApps they are not real vCloud Director vApps until they are adopted. The adoption happens when the VM inside the vApp is somehow reconfigured.

By default VM discovery is enabled for every Organization in vCloud Director. It can be disabled in General Settings (UI or API)

general-settings

or with CMT command on VCD cell:

cell-management-tool manage-config -n managed-vapp.discovery.activated -v false

This behavior can be overridden at Org VDC level with API element <VmDiscoveryEnabled>.

Here are the differences between Discovered and Adopted vApp:

Discovered vApp

  • Looks like a regular vApp
  • Can have only one VM per discovered vApp
  • vApp contains API element <autoNature>true</autoNature>
  • When imported VM is deleted in VC or VCD, its vApp object will get automatically purged
  • Is owned by system
  • Is not subject to Org lease settings

Adopted vApp

  • Regular vCloud Director vApp
  • Can contain multiple VMs, vApp networks, etc.
  • Discovered vApp will get adopted when it is reconfigured (other than changing its name or description)

Other Considerations

  • Discovery process runs in the background every 3 mins
  • Failed VM import is retried after 60 mins. This can be changed with CMT command (example for 25 seconds):
    cell-management-tool manage-config -n managed-vapp.discovery.retry-delay-sec -v 25
  • The following VMs cannot be imported: Fault Tolerant VMs, VMs with creation/upload process in VC, templates, VCD shell VMs
  • VM must be connected to Org VDC network.
  • VM can be running or powered off.
  • VM does not need to use Org VDC storage policy. If it resides on unknown storage policy, it is automatically relocated to the default Org VDC storage policy during the adoption however the VM must be in powered off state.
  • VM with IDE controller must be in powered off state.
  • VM CPU/RAM resources are changed based on Org VDC allocation type.
  • VM resources are not subject to Org VDC allocation restrictions, but are charged against it.
  • VM name in VC remains intact until it is adopted and renamed
  • New vSphere 6.5 guest operating systems are not recognized and are imported as Other (32-bit) OS.

Update 4/24/2017

  • By default there is minimal VM age configured to 1 hour. It means VMs that were freshly reconfigured will be skipped for the import. This is to ‘settle’ the VMs first. The interval can however be changed with CMT command (age set to 60 seconds):
    cell-management-tool manage-config -n VM_DISCOVERY_MIN_AGE_SEC -v 60
  • Related to the minimal age, it is important that all cells and vCloud database are using the same time configuration. Not only the time must be correct but also the time zone must be configured identically.

Monitoring vSphere Replication RPO Compliance

Just a quick post to show how you can monitor Recovery Point Objective (RPO) compliance of a virtual machines protected with vSphere Replication.

Option 1: vCenter Server Alarm

When vSphere Replication Appliance is registered to vCenter Server multiple new vSphere Replication Event Types become available and can be used for creation of custom alarms.

List of all these event types can be queried with the following one-line PowerCLI command:

(get-view eventManager).get_Description()| select -expand Eventinfo |where FullFormat -like “*Hms*”

The following example will show how to set alarm for event “RPO violated”

Key:ExtendedEvent
Description: RPO violated
Category: error
FullFormat: com.vmware.vcHms.rpoViolatedEvent|Virtual machine vSphere Replication RPO is violated by [data.currentRpoViolation] minute(s)

  1. In vCenter Server go to Manager, Alarm Definitions and add new alarm
  2. Set alarm name, monitor VMs and specific events.
    new-alarm
  3. Enter the trigger (com.vmware.vcHms.rpoViolatedEvent)
    alarm-trigger
  4. Add Alarm actions (email, SNMP trap, run command etc.) as necessary.

Triggered alarm:

triggered-alarm

Note that this alarm applies only to VMs replicated from the particular vCenter Server. So it will not be triggered on VMs replicated to this vCenter Server.

Option 2: vCloud API

This options applies only for VM replications to or from a cloud provider who uses vCloud Availability add-on. The vCloud Director tenant APIs are extended with replication APIs. The state of each replication can be retrieved with:

GET /api/vr/replications/<replication-id>

and

GET /api/vr/failbackreplications/<replication-id>

Where list of all replications and their replication-ids is retrieved at org level with these two API calls:

GET /api/org/<org-id>/replications

and

GET /api/org/<org-id>/failbackreplications

An example of VM1 replication state (RPO 15 mins, not active with 16 min RPO violation):

replication-api

The following tables describes all the elements of the API response:

replication-details