vCloud Director vApp Runtime Lease Expiration Action

In vCloud Director it is possible to configure vApp leases. The maximums are set by system admin at Organization level (in Policies), which can be lowered by Org Admin (at org level) and set by vApp owner at the vApp level. A vApp has runtime lease (for how long it will be in running state) and storage lease (for how long it will consume storage once it is not running).

vApp leases are very useful in test & dev or lab environments to make sure abandoned, unused VMs are not running and taking resources.

When vApp lease is coming to an end, its owner gets a reminder via email (how many days before expiration can be configured in User Preferences) and can optionally reset vApp lease to avoid its stopping or deletion.

By default expired running vApp is put into suspended state which means its memory content is saved to datastores. This ensures fully consistent state upon consequent power on of the vApp. This however make not be always needed especially in dev/lab situations – the memory content could take lots of storage space and for example saving 16 GB RAM VM to datastore could also create IO performance impact. As of vCloud Director 8.20 the Organization Administrator can instead change the default runtime expiry action to power off. The setting is done at Org level and must be done via API by setting the element <PowerOffOnRuntimeLeaseExpiration> of OrgLeaseSettingsType to true. The API version must be at least 25.0.

PUT /api/admin/org/eea1f10c-3fee-43d7-bd8e-be63453d6e34/settings/vAppLeaseSettings


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<VAppLeaseSettings xmlns="">


When the vApp expiry action is set to power off, the actual VM stop action power off (hard) vs shutdown (gracefull) procedure depends on the vApp’s config for each VM (tab Starting and Stopping VMs).

Also note that subsequent edit of Org policies in UI will reset the Org PowerOffOnRuntimeLeaseExpiration setting back to default (false).


vSphere Replication Issue with ESXi 6.5U1

This is a quick post to highlight an issue vSphere Replication has with ESXi 6.5U1 for To-the-cloud replication.

Only customers that use vSphere Replication for DR or migrations to the cloud endpoints (e.g. vCloud Availability for vCloud Director) with ESXi 6.5U1 hosts are affected (ESXi 6.5 and older works fine). Also host-to-host replication is not affected.

The root cause is that ESXi 6.5U1 hosts are unable to retrieve from vSphere Replication Appliance vr2c-firewall.vib that is responsible for opening outgoing communication ports for replication traffic on the ESXi host firewall.

This results in inability to perform any to-the-cloud replications. To see the issue look into the host Firewall configuration in the Security Profile section. If you do not see Replication-to-Cloud Traffic section you are affected.

The picture below which traffic it is related to (red rectangle on the left):

If you would look into esxupdate.log on the host you will see error: [Errno 14] curl#56 – “Content-Length: in 200 response”.

Until a fix is going to be released here is a workaround:

  1. Download the vr2c-firewall.vib from the vSphere Replication Appliance: https://vSphere-Replication-Appliance-ip-or-fqdn:8043/vib/vr2c-firewall.vib.
  2. Upload the vib to a shared location (datastore)
  3. Install the vib to every host with the following command: esxcli software vib install -v /vmfs/volumes/<datastore>/vr2c-firewall.vib
  4. Verify the fix was installed properly with: esxcli software vib list | grep vr2c

Missing Licensing Metrics in vCloud Director 9.0

You might have noticed that vCloud Director 9.0 no longer displays licensing metrics.

This was a conscious decision as already for some time the licensing is handled externally by vCloud Usage Meter and the metering in vCloud Director caused vCloud database bloat.

If for some reason you still need to have these metrics available in UI, vCloud API or vROps Management Pack you can enable license metering with this command on a cell. No need for reboot, just wait few minutes for the next data collection.

$VCLOUD_HOME/bin/cell-management-tool manage-config -n licensing.metrics.vm.enabled -v true

How to Configure Additional VM Metrics in vCloud Director

vCloud Director already for some time (since version 5.6) provides to tenants basic set of VM metrics. Until vCloud Director 9.0 they had to be retrieved with vCloud API, however now the users can easily access the metrics from the new HTML5 UI.

vCloud Director 9.0 besides adding the metric UI also simplifies the metric database backend configuration (no KairosDB needed anymore) and also provides the option to service provider to configure additional VM metrics.

Here follows the step by step description of the last point. I assume that Cassandra cluster – the VM metric database is already set up.

  • First you need to find the metric names VC Performance Manager uses. This PowerCLI script exports all VM metric names.
  • Now we will create text file (e.g. metrics.groovy) with the new metrics.

configuration {

The metrics in the file must not overlap with the already existing default metrics. Additional options about frequency, interval averages, etc. can be provided as well. See docs for the details.

  • On a vCloud Director cell with the cell-management-tool we will import the new metrics:
    $VCLOUD_HOME/bin/cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy 
  • Still on the cell we need to update Cassandra schema, again with the cell-management-tool (provide the correct nodes addresses, DB authentication details, port and metrics time to live in days):
    $VCLOUD_HOME/bin/cell-management-tool cassandra –configure –cluster-nodes –username cassandra –password cassandra –port 9042 –ttl 31 –update-schema
  • Restart all cells

That is all. After while we can monitor the new metrics with the UI or API.

The metric definition is stored in vCloud Director table metric_configuration.

vCloud Availability: Replication of Powered-off VM

Just a short post about a feature I recently learned.

In vSphere Replication when you are configuring replication of powered-off VM you will get the following message:

The virtual machine is not powered on. Replication will start when the virtual machine is powered on.

The replication is actually configured and its placeholder VM is created in the recovery location (cloud) but the VM will stay in Not Active state.

Why is this? Immediate start of replication locks VM disks which means such VM would not be able to power-on until the initial sync is finished. But what if you want to replicate powered-off VMs for example templates that are never meant to run?

You can in fact force start the replication by right clicking the VM and selecting Sync Now, which asks confirmation question if we really want to do so as the VM will not be able the be powered on until the operation completes.

Is there a use case for this? As I mentioned this could be used for catalog sync as replication is much faster and efficient that OVF export / import.