vRealize Automation with Multiple Cloud Endpoints

One of my customers had deployed true hybrid vRealize Automation with multiple cloud endpoints: vCloud Air and internal vCloud Director and AWS. I was called in to troubleshoot strange issue where sometimes deployment of a cloud multimachine blueprint (vApp) would work but most often it would fail with the following message:

VCloud Clone VM failed for machine: XXX100 [Workflow Instance Id=19026]
System.InvalidOperationException: Error occurred while getting vApp template with ID: urn:vcloud:vapptemplate:a21de50d-8b5e-41a6-81d1-acfd8ab8364b

INNER EXCEPTION: com.vmware.vcloud.sdk.utility.VCloudException: [ 8ae6fbca-e0d2-43e7-bc94-5bc9d776bf8d ] No access to entity “com.vmware.vcloud.entity.vapptemplate:a21de50d-8b5e-41a6-81d1-acfd8ab8364b”

Endpoint was properly configured, template existed, so what could be wrong? Why were we denied the access to the template?

It turns out that by design vRealize Automation does not match a template to a particular endpoint. It identifies it just by name. So in our case sometimes it would try to deploy the blueprint to wrong endpoint where the template of the particular name did not exist.

The fix is simple:

  • Define reservation policies which would identify each endpoint.
  • Assign them to the proper reservationsReservation
  • Assign reservation policies to the Cloud vApp blueprint. This way there will never be confusion from which template to provision to which endpoint.Blueprint reservation policy

vCloud Connector and Offline Data Transfer

Offline Data Transfer (ODT) is a feature of vCloud Connector that allows migration of VMs from customer own datacenter to vCloud Air with NAS appliance which is shipped via regular mail. The point is to avoid slow wide area network connectivity and leverage awesome bandwidth but slow latency of sneakernet.

Have you ever wondered why it is supported only with vCloud Air and not with any public or private cloud based on vCloud Director? Well I am going to lay down the whole process here in this blog post so nothing is stopping anyone testing this feature on your own.

Let me first paste picture from the manual which describes in high level how the process works:

Offline Data Transfer ProcessvCloud Connector (vCC) is leveraged to manage the whole process. The customer (on left) deploys his own vCloud Connector Server and Node which he attaches to his on premises infrastructure (vSphere based). He then requests the ODT service. The provider will deploy ODT node in the public cloud (on right) and also its own vCC Server to manage it. Regular NAS appliance is prepared – its only purpose is to provide storage capacity which is fast and reliable enough via NFS protocol and can be easily packaged and shipped.

Customer mounts it to his vCC Node (to a directory via NFS mount). Both the ODT and vCC Nodes are registered in his vCC Server. Then via the traditional vSphere Client and vCC Plugin only the local vSphere environment (here it differs from the traditional vCC transfer).

vSphere in vCC

The actual export is done by selecting the objects to export (templates, vApps or VMs) and clicking the small Offline Data Transfer icon: ODT IconMount path is entered and links and credentials to the target Cloud and ODT node. There is also option to select if a particular VM should be deployed and connected to a network. These steps above are all described in the manual here.

But what about the provider side of the whole process?

ODT Node

ODT Node is actually regular vCloud Connector Node tweaked by running configureSneakernetNode.sh script which can be found in /opt/vmware/hcagent/scripts folder on the Node VM itself.

The ODT needs to have network access to the vCenter Server (and ESXi hosts) of the target vCloud VDC environment.


The actual import is done via provider vCloud Connector server which is again the regular vCloud Connector server with no tweaks this time. The ODT Node is registered there which enables import menu in the vCC plugin GUi in vSphere Client. The shipped NAS appliance must be mounted to the ODT Node and the ODT URL and mount path is entered in the Import Wizzard. The actual physical connection of the NAS appliance can be done using dedicated VLAN with point to point connection of the second ODT network interface.

ImportNext we need to pick the target vCenter Server and a credentials for it. ODT Node will import offline VMs which are stored as encrypted OVFs on the NAS appliance into the target vCenter Server. To do that it needs a big enough datastore and a dummy network in order to connect the imported VMs temporarily to it. Once that is done VMs are imported by vCloud Director to the target VDCs, catalogs and networks. The provider needs to have big enough datastore and create dummy standard switch port group on every host with name ‘VM Network’. This network does not need to have external access.

As you can see contrary to the regular internet vCloud Connector transfer where the VM is transfer from the original environment via on-prem node to public node to vCloud Director (through its API and transfer storage – see here for more detail) the transfer does not go through the vCloud Director cells and its transfer storage at all. This is possible thanks to handling the final step of the process by the provider himself (he has vCenter Server access) and makes also the transfer faster (potentially one less step). On the other hand this brings some security and operational process challenges (physical access to management network, vCenter credentials) which must be properly addressed.

VCD Cell Management Tool without Administrator Credentials

I just learned from engineering neat trick related to how cell management tool can be invoked without specifying administrator credentials.

The issue is that currently you cannot use LDAP account to trigger cell management tool commands which are mostly used for quiescing and shutting down cells for maintenance. Using vCloud Director local administrator account is discouraged as it poses a security issue. However what is possible is to trigger the cell management tool as root (or with sudo) and supply via hidden flag -i the process ID of the java process.

Here is an example:


First I query the java PID with ps aux command. Then I use the standard cell-management-tool command without specifying the user with the -i flag at the end.

So you can force the administrator to log in to the cell guest OS via a LDAP account and then run the command with sudo.

Thank you Zachary Shepherd for the tip.

Custom vCenter Server Event and Alarm

Related to my previous post about monitoring Edge Gateways my customer asked me if he could leverage vCenter Server alarms as they are integrated with their monitoring and alerting infrastructure.

So basically is there a way to create vCenter alarm via scripted action (for example with PowerCLI)?

The answer is yes and it is not that difficult. There are two types of vCenter alarms: based on condition/state or an event. And it is possible to create custom user event via Loguserevent method of entity manager via vSphere API.

This is example of PowerCLI code the creates user event “Edge Gateway event” at the root datacenter folder.

$DCFolderEntity = Get-Folder -Name datacenters
$eventMgr = Get-View EventManager
$eventMgr.LogUserEvent($entity.ExtensionData.MoRef,"Edge Gateway event")

User logged event

Now it is easy to create custom alarm based on the user logged event. Create the alarm at the vCenter Server root level, as alarm type pick monitor vCenter Server and as triggers manually enter (type):

Event; vim.event.GeneralUserEvent 

That’s it.