vSphere Replication Issue with ESXi 6.5U1

This is a quick post to highlight an issue vSphere Replication has with ESXi 6.5U1 for To-the-cloud replication.

Only customers that use vSphere Replication for DR or migrations to the cloud endpoints (e.g. vCloud Availability for vCloud Director) with ESXi 6.5U1 hosts are affected (ESXi 6.5 and older works fine). Also host-to-host replication is not affected.

The root cause is that ESXi 6.5U1 hosts are unable to retrieve from vSphere Replication Appliance vr2c-firewall.vib that is responsible for opening outgoing communication ports for replication traffic on the ESXi host firewall.

This results in inability to perform any to-the-cloud replications. To see the issue look into the host Firewall configuration in the Security Profile section. If you do not see Replication-to-Cloud Traffic section you are affected.

The picture below which traffic it is related to (red rectangle on the left):

If you would look into esxupdate.log on the host you will see error: [Errno 14] curl#56 – “Content-Length: in 200 response”.

Until a fix is going to be released here is a workaround:

  1. Download the vr2c-firewall.vib from the vSphere Replication Appliance: https://vSphere-Replication-Appliance-ip-or-fqdn:8043/vib/vr2c-firewall.vib.
  2. Upload the vib to a shared location (datastore)
  3. Install the vib to every host with the following command: esxcli software vib install -v /vmfs/volumes/<datastore>/vr2c-firewall.vib
  4. Verify the fix was installed properly with: esxcli software vib list | grep vr2c

Advertisements

Missing Licensing Metrics in vCloud Director 9.0

You might have noticed that vCloud Director 9.0 no longer displays licensing metrics.

This was a conscious decision as already for some time the licensing is handled externally by vCloud Usage Meter and the metering in vCloud Director caused vCloud database bloat.

If for some reason you still need to have these metrics available in UI, vCloud API or vROps Management Pack you can enable license metering with this command on a cell. No need for reboot, just wait few minutes for the next data collection.

$VCLOUD_HOME/bin/cell-management-tool manage-config -n licensing.metrics.vm.enabled -v true

How to Configure Additional VM Metrics in vCloud Director

vCloud Director already for some time (since version 5.6) provides to tenants basic set of VM metrics. Until vCloud Director 9.0 they had to be retrieved with vCloud API, however now the users can easily access the metrics from the new HTML5 UI.

vCloud Director 9.0 besides adding the metric UI also simplifies the metric database backend configuration (no KairosDB needed anymore) and also provides the option to service provider to configure additional VM metrics.

Here follows the step by step description of the last point. I assume that Cassandra cluster – the VM metric database is already set up.

  • First you need to find the metric names VC Performance Manager uses. This PowerCLI script exports all VM metric names.
  • Now we will create text file (e.g. metrics.groovy) with the new metrics.

configuration {
metric("mem.granted.average")
metric("mem.granted.maximum")
}

The metrics in the file must not overlap with the already existing default metrics. Additional options about frequency, interval averages, etc. can be provided as well. See docs for the details.

  • On a vCloud Director cell with the cell-management-tool we will import the new metrics:
    $VCLOUD_HOME/bin/cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy 
  • Still on the cell we need to update Cassandra schema, again with the cell-management-tool (provide the correct nodes addresses, DB authentication details, port and metrics time to live in days):
    $VCLOUD_HOME/bin/cell-management-tool cassandra –configure –cluster-nodes 172.16.0.41 –username cassandra –password cassandra –port 9042 –ttl 31 –update-schema
  • Restart all cells

That is all. After while we can monitor the new metrics with the UI or API.

The metric definition is stored in vCloud Director table metric_configuration.

vCloud Availability: Replication of Powered-off VM

Just a short post about a feature I recently learned.

In vSphere Replication when you are configuring replication of powered-off VM you will get the following message:

The virtual machine is not powered on. Replication will start when the virtual machine is powered on.

The replication is actually configured and its placeholder VM is created in the recovery location (cloud) but the VM will stay in Not Active state.

Why is this? Immediate start of replication locks VM disks which means such VM would not be able to power-on until the initial sync is finished. But what if you want to replicate powered-off VMs for example templates that are never meant to run?

You can in fact force start the replication by right clicking the VM and selecting Sync Now, which asks confirmation question if we really want to do so as the VM will not be able the be powered on until the operation completes.

Is there a use case for this? As I mentioned this could be used for catalog sync as replication is much faster and efficient that OVF export / import.

How To Disable Local System Administrator Accounts in vCloud Director

For some time there has been a hidden security feature in vCloud Director that allows disabling local system administrator accounts. During vCloud Director installation a default local system administrator account is created. The user credentials are stored encrypted in the vCloud Director database but there is no way to enforce complex password policies other than Account Lockout Policy.

It is possible to configure external identity sources such us generic LDAP for basic authentication and SAML2 IdP (such as vCenter SSO). The authentication and thus also the password policies are than managed externally. However, when you try to delete or disable all local system administrator accounts you will get the following error:

Cannot delete or deactivate the last system administrator.

This is a built in protection against completely locking yourself out when the external identity sources are not available.

Some customers can have the need to enforce strict security rules on all vCloud Director system administrator logins. There is a non-documented way to disable all local system administrator accounts with a single command. The system administrator can run the following cell-management-tool  command to enable config property local.sysadmin.disabled.

$VCLOUD_HOME/bin/cell-management-tool manage-config -n local.sysadmin.disabled -v true

Immediately after the property is enabled, authentication with local accounts will stop working. It means authentication for all local system administrator accounts that exist in vCloud Director (not just the default account created during installation) will be rejected. Organization local accounts will not be affected.

In case access to external IdPs is lost, the system admin can again disable the property to regain access to vCloud Director:

$VCLOUD_HOME/bin/cell-management-tool manage-config -n local.sysadmin.disabled -v false