What’s New in Cloud Director 10.1

And it is time for another What’s New in (v)Cloud Director blog post. If you are not up to date you can find the older articles for versions 10, 9.7, 9.5 and 9.1 here.

Let us start with the “important” announcement – a name change. vCloud Director has been re-branded to VMware Cloud Director. Fortunately we keep the same (unofficial) acronym – VCD. The current version is 10.1 which might look like a small increase from the previous 10.0 but that is just marketing numbering so do not put too much emphasis on it and assess yourself if it is big release or not.


VMware has added support for NSX-T 3.0, but vSphere 7 support is missing. It is expected to come shortly in a major patch release (correction, NSX-T 3.0 did not make it yet too). You can upgrade your management clusters and dedicated vCenter Servers, just not those that are backing Provider VDCs.


As previously announced, no more Adobe Flex UI (cannot be even enabled with a secret switch). Shouldn’t be an issue however, as the HTML 5 UI has not only 99.9% parity but in fact is significantly better than Flex UI ever was. There are new features such as VM sizing and VM placement groups, advanced filtering, multiselect actions, badges, quick access to VM console or network cards, tasks and events in vApp details, …

The UI team is no longer in feature parity mode, they are in make-it-better mode and doing a great job as can be seen from the screenshots below.

Platform Security

Certificate validation is now required for VC/NSX endpoints and will be required for LDAP in the next major release as well. It means your endpoints either have to have publicly trusted certificates, or you have to upload their signing certificate, or you have to approve their certificate on the first use (when you add or edit such endpoint). To ease the transition from older vCloud Director releases, you can run cell-management-tool trust-infra-certs command after upgrade that will automatically retrieve and trust certificates of all connected VC/NSX endpoints. If you forget to run this command, Cloud Director will not be able talk to VC/NSX endpoints!


While you still can use the Linux installer of Cloud Director with external database the appliance deployment factor has been again improved and there is less and less reasons not to use it especially for green field deployments.

The appliance API (on port 5480) has been enhanced to monitor status of services, to see which node is running active node of embedded database (useful for load balancing access to the database for external tools), monitor filesystem storage and trigger database node switchoever (planned failover) or promotion (after failure).

The appliance embedded appliance has for the first time automated failover functionality. It is disabled by default but you can enable it with the API.The Appliance UI has also been improved and provides some of the API functionality.

Messaging Bus

As you might now, Cloud Director has embedded messaging bus for inter-cell communication. In the past it was using ActiveMQ (ports 61611 and 61616). If the service provider wanted to use blocking tasks, notifications or vCloud API (or is it now VCloud API?!) extensions then an external RabbitMQ messaging bus had to be deployed. In the current release ActiveMQ has been replaced with Artemis and is also available externally for blocking tasks and notifications, so RabbitMQ is no longer needed for these two use cases. Additionally it can also be used by tenants.

Artemis uses MQTT communication protocol and the connection to it can be made via WebSocket session which is authenticated with regular Cloud Director authentication token mechanism.

Note that external RabbitMQ is still supported and still needed for vCloud API extensibility use case.


NSX-T integration has been enhanced with routed Org VDC networks (previous release supported NAT-routed) with BGP routing protocol. This feature currently requires dedicated Tier-0 Gateway for the tenant.
IPSec VPN is now in the UI (pre-shared key authentication and policy based are supported). IP Sets and Security Groups have been split with support for network objects that dynamically refer to all connected VMs.

The service provider can configure multiple NSX-T Edge Clusters and select which one will be used for a particular Tier-1 (Org VDC) Gateway. This enables separation of Tier-0 and Tier-1 Gateways to different Edge Clusters.

The NSX-V side of networking has also one new feature – you can now use Cross VDC networking within the same vCenter Server. This for example enables multi egress networks within single Org VDC for stretched cluster use case.


vSphere VM Encryption is now supported within Cloud Director. The encryption happens in the hypervisor which means the data is encrypted both in rest as well in flight. The encryption is set up via vCenter Server storage policies by enabling host based rules. A compatible external key management server must be deployed and connected to vCenter Server. It means the feature is fully in realm of the service provider and key management is not exposed to tenants.

Other Features

  • Proxying of dedicated vCenter Servers (so called Centralized Point of Management – CPoM feature) was improved with extra stats, more proxies and browser extension to simplify the usage
  • Support for VM (UI) and vApp (API only) live migration between Provider VDCs
  • Due to UI upgrade to Clarity 2+ custom themes will have to be recompiled
  • The provider can enable promiscuous mode and forged transmits on VXLAN backed Org VDC network (API only)
  • Blocking tasks for OpenAPI tasks.
  • Cloud Director 10.1 is the first release that enables automated NSX-V to NSX-T migration. More on that in a later blog post.

vSAN File Services with vCloud Director

vSphere 7 is now generally available and with it came also new vSAN update that introduces vSAN File Service. Cormac Hogan has good overview of the feature on his blog so definitely head there first to understand what it is.

I want to dive into the possibility of using vSAN File Service NFS in vCloud Director environments.

Let me start with current (April 2020) interop – vSphere 7 is not supported yet with vCloud Director. Which means vCenter Server 7 cannot be used as a target for IaaS services. But that is not an issue for the use case I want to discuss today.

vCloud Director deployment needs NFS storage for its Transfer Share directory. vCloud Director architecture consists of multiple cells (VM nodes) that scale out horizontally based on the size of the managed environment. The cells need shared database and shared Transfer Share directory to function properly. The Transfer Share must be NFS mount and is used mostly for OVF import/export operations related to vApp template and catalog management however the appliance deployment mode of vCloud Director also uses transfer share for storing appliance related info, ssh keys, responses.properties file for deployment of additional cells, and embedded database backups.

vCloud Director cell VMs are usually deployed in the management cluster and that can be separate vSphere 7 environment with vSAN. Can we (or should we) use vSAN NFS for vCloud Director Transfer Share?

Current practice is either to use external hardware storage NFS (NetApp) or to deploy Linux VM with large disk that acts as NFS server. The first approach is not always possible especially if you use vSAN only and have no external storage available. Then you have to go with the Linux VM approach. But not anymore.


vSAN File Service NFS has the following advantages:

  • no external dependency on hardware storage or Linux VM
  • easy to deploy and manage from UI or programmatically
  • capacity management with quotas and thresholds
  • high availability
  • integrated lifecycle

The whole end-to-end deployment is indeed very simple, let me demonstrate the whole process:

  1. Start with vSAN FS configuration in vSphere Cluster > Configure > vSAN > Services > File Service
  2. Directly download vSAN File service agent (the lightweight container image OVA)
  3. Configure vSAN domain and networking

  4. Provide pool of IP addresses for the containers (I used 4 as I have 4 host management cluster).
  5. After while you will see the agent containers deployed on each cluster node.
  6. Now we can proceed with NFS share configuration. In the vSphere Cluster > Configure > vSAN > File Service Shares > ADD. We can define name, vSAN storage policy and quotas.
  7. Enter IP addresses of your vCloud Director cells to grant them access to this share. Set permission Read/Write and make sure root squash is disabled.
  8. Once the share is created, select the check box and copy its URL. Chose the NFSv4.1.
    In my case it looks like
  9. Now you use the string in your vCloud Director cell deployment. I am using the vCloud Director appliance.
  10. Once the cell is started we can see how the transfer share is mounted:

    Notice that while the mount IP address in /etc/fstab is the provided one, the actual one used is This provides load balancing across all service node when more exports are created and when NFSv4.1 mount address is used.

It is imported to understand that although we have 4 vSAN FS agents deployed, the TransferShare will be provided via single container – in my case it is the one with IP To find out on which host this particular container is running go to Cluster > Monitor > vSAN > Skyline Health > File Service > File Service Health.

So what happens if the host esx-03a.corp.local is unavailable? The share will fail over to another host. This took in my tests around 60-90 seconds. During that time the NFS share will not be accessible but the mount should persist and once the failover finishes it will become accessible again.

Notice that is now served from esx-04.corp.local.

Also note that maintenance mode of the host will not vMotion the agent. It will just shut it down (and after while undeploy) and rely on the above mechanism to fail over the share to another agent. You should never treat the vSAN FS agents as regular VMs.

I am excited to see vSAN File Services as another piece of VMware technology that removes 3rd party dependencies from running vCloud Director (as was the case with NSX load balancing and PostgreSQL embedded database).

PowerCLI: Open vCloud Director VM in Standalone VM Console

A short Powershell script I wrote to demonstrate how to open vCloud Director VM console in the (Windows) Standalone VM Console.

$vm = get-CIVM 'CentOS7'
$vmrcPath = "C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe"
$mks = $vm.ExtensionData.AcquireMksTicket()
$thumbprint = $mks.Ticket.Substring($mks.Ticket.get_Length()-61,59)
$parameter = "vmrc://$($mks.Host)/?thumbprint=$($thumbprint)&path=$($mks.Vmx)&websocket=wss://$($mks.Host)/($mks.Port);$($mks.Ticket)"
& "$vmrcPath" $parameter

The script should be pretty self explanatory. You need to have PowerCLI for vCloud Director installed and be logged in (as tenant or system admin).

$vm variable contains the VM for which you want to open the console. $vrmcPath is the path to the locally installed VMware Remote Console application. Next you need to acquire VM’s MKS ticket and with little manipulation send it to the application as a parameter.

Automate Let’s Encrypt Certificates – Part 2

Some time ago I blogged about how I automate acquisition of Let’s Encrypt Certificates for my lab (NSX + vCloud Director) with PowerShell. The old script no longer works due to some changes on Let’s Encrypt side therefore the need for part 2.

To quickly summarize my situation. My lab consists of vCloud Director with multiple cells fronted by NSX-V Load Balancer. I need public certificate for vCloud Director which is uploaded to the NSX-V Load Balancer (that does L7 SSL termination) and to vCloud Director public addresses.


  • Web server on the domain your are getting the certificate for. It is necessary for the DNS challenge that proves you own the domain you are requesting the certificate for. I am using IIS on the machine I trigger the script from and supply the root folder where the challenge file needs to be placed.
  • NSX-V API access information – needed to replace the certificate on the NSX-Edge
  • Details about the load balancer (on which Edge it is running and what is the LB application profile of vCloud Director)
  • vCloud Director API access information – needed to upload new certificate and the full chain to vCloud Director public addresses.
  • PowerShell modules: POSH-ACME and PowerCLI

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:admin@fojta.com"
$Domain = "vcloud.fojta.com"
$Vcd = "vcloud.fojta.com"
$VcdAdmin = "administrator"
$VcdPassword = "vcloud"
$IisAcmeRoot = "C:\inetpub\wwwroot\.well-known\acme-challenge"

$RootCert = "-----BEGIN CERTIFICATE-----

#Set-PAServer LE_STAGE
Set-PAServer LE_PROD

## Read https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

New-PAAccount -AcceptTOS -Contact $Email
New-PAOrder $Domain

$auths = Get-PAOrder | Get-PAAuthorizations
$token = $auths[0].HTTP01Token
$toPublish = Get-KeyAuthorization $token

## Upload challenge file to the IIS web server
New-Item -Path $IisAcmeRoot -Name $token -Value $toPublish

$auths.HTTP01Url | Send-ChallengeAck
New-PACertificate $Domain
$cert = Get-PACertificate

$IssuerCert = [IO.File]::ReadAllText($cert.ChainFile)
$PrivateKey = [IO.File]::ReadAllText($cert.KeyFile)
$LBCertificate = [IO.File]::ReadAllText($cert.CertFile)

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<pemEncoding>" + $LBCertificate + $IssuerCert + $RootCert + "</pemEncoding>
<privateKey>" + $PrivateKey + "</privateKey>
<description>vCloud Certificate</description>
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Delete Root and intermediate certificate from the Edge as they are not needed
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[0]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[1]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId[2]
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Update vCloud Director with new certificates

$VcdSession = Connect-CIServer $Vcd -User $VcdAdmin -Password $VcdPassword

$Uri = "https://"+$Vcd+"/api/admin/extension/settings/general"
$head = @{"x-vcloud-authorization"=$VcdSession.SessionSecret} + @{"Accept"="application/*;version=33.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$sxml.GeneralSettings.RestApiBaseUriPublicCertChain = $LBCertificate + $IssuerCert + $RootCert
$sxml.GeneralSettings.SystemExternalAddressPublicCertChain = $LBCertificate + $IssuerCert + $RootCert

$r = Invoke-WebRequest -URI $Uri -Method Put -Headers $head -ContentType "application/vnd.vmware.admin.generalSettings+xml" -Body $sxml.OuterXML -ErrorAction:Stop

vCloud Director with TLS-only Connection with External Database

Very brief blog post to document how to install vCloud Director with external database that does not support plain text connections.

In general the process is to do the initial set up with plain text DB connection and then switch to TLS – see the official docs here. That will however not work if the external database supports only TLS connection.

Instead this process must be used:

  1. Import DB certificate (unless it is publicly signed) to cell default JRE keystore.
  2. Use unattended configuration.


# /opt/vmware/vcloud-director/jre/bin/keytool --import -trustcacerts -keystore /opt/vmware/vcloud-director/jre/lib/security/cacerts -alias psql -file /opt/vmware/vcloud-director/etc/psql.crt


Enter keystore password: changeit


Owner: CN=

Issuer: CN=

Serial number: cb64ae0954184182

Valid from: Fri Nov 22 14:10:39 GMT 2019 until: Sat Nov 21 14:10:39 GMT 2020 Certificate fingerprints:

MD5:  04:4F:8F:C5:9C:CC:D5:E8:F1:50:C1:85:51:D4:FB:AD

SHA1: 22:53:FF:71:A7:EC:9B:D1:74:79:D5:95:46:71:F6:38:A7:E7:F8:4E

SHA256: 08:7C:27:B4:FB:32:04:DE:AF:BB:FE:9D:47:1D:38:46:C8:F4:7C:69:73:DE:8D:CB:BD:2A:A5:B2:11:12:68:DD

Signature algorithm name: SHA256withRSA

Subject Public Key Algorithm: 2048-bit RSA key

Version: 3




#1: ObjectId: Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n




#2: ObjectId: Criticality=false BasicConstraints:[





#3: ObjectId: Criticality=false SubjectKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n




Trust this certificate? [no]:  yes

Certificate was added to keystore



# /opt/vmware/vcloud-director/bin/configure --unattended -dbhost <DB IP address> -dbname vcloud -dbpassword vcloud -dbtype postgres -dbuser vcloud --database-ssl true –dbport 5423 -ip <cell-ip> --primary-port-http 80 --primary-port-https 443 -cons <cell-ip> --console-proxy-port-https 8443 -k /opt/vmware/vcloud-director/etc/certificates.ks -w <keystore password> -g


Database configuration complete.


# /opt/vmware/vcloud-director/bin/cell-management-tool system-setup --email admin@vcloud.com --full-name 'System Admin' --installation-id 33 --password 'VMware1!' -system-name vcd --unattended --user administrator

Creating admin user.

Setting system details.

Completing system setup.

System setup is complete.