What’s New in vCloud Director 9.7

With an impressive cadence after less than 6 months there is a new major release of vCloud Director – version 9.7. As usual, I will go into the new features from the technical perspective and provide additional links to related blog post in the future.

Just for reminder, you can find older What’s New in vCloud Director blog posts here: 9.5, 9.1.

Tenant User Interface Evolution

The legacy Flex UI is still here, but there is less and less reasons to use it. My guess is that 95% of the Flex features are ported to the HTML5 UI which also provides additional exclusive feature.

  • Branding: the service provider can now (with CloudAPI) change color scheme, theme, logos (including favicon and login page) and title not only globally but also individually for each tenant.  The provider can also add structured custom links and override existing links (help, about and standalone VMware Remote Console download). The custom links can also be dynamic with inclusion of (self-explaining) custom variables: ${TENANT_NAME}, ${TENANT_ID} and ${SESSION_TOKEN}
  • New ribbon offers quick glance on the content of all Organizations that logged in user has access to.
  • Recent Tasks pane provides immediate info about what is going on (last 50 items in past 12 hours)
  • Global search provides quick way to find particular object across VDCs or even sites:
  • vApp Network Diagram now shows vApp logical networking
  • Besides these large changes there are many small enhancements that level up tenant UI to nearly fully cover the legacy Flex UI. Users are actively encouraged to start using the Tenant UI with the yellow banner on top legacy UI:

Service Provider Admin UI

  • Service Provider Admin HTML5 UI adds access to Cloud Resources, vSphere Resources, Blocking Tasks to continue the process of adding features available from Admin Flex UI. Some items are still read only. On the other hand some (new) features are only available in the H5 UI: for example adding NSX-T Manager, Flex allocation model, etc.

New Compute Features

Flexible allocation model

The service provider can create (H5 UI only) completely new type of Org VDC – Flexible Allocation Model. The new model actually covers all the legacy allocation models (see here) plus provider can define completely new way for VMs and VDC to consume vSphere compute resources.

It is also possible to change allocation model of existing Org VDCs. For now that feature is possible only for Org VDCs created in 9.7 release though.

Compute Policies

While compute policies were already introduced in the previous release, the feature functionality is now enhanced and additionally simplified by including the tenant part in the H5 UI.

They are used to control resource allocation for example for licensing, performance or availability use cases. Provider defines (via vCloud Open API) policies that the tenant can then assign to deployed VMs.

Provider can for example define Microsoft Windows licensed hosts, create appropriate policy and assign it to Windows templates. Any VM deployed from such template will be placed only on those hosts.

Similarly provider can define high performance compute policy, which results in higher VM’s CPU limit and reservation. Tenant can chose and apply such policy for subset of her workloads.

Tenant could also use this feature to select site deployment for each particular VM for Org VDCs backed by vSphere Metro Cluster.

New Networking Features

Edge Cluster

The service provider has now the ability to control placement of each Org VDC Edge Gateway node (both for compute and storage) in order to get better resiliency and to secure higher SLAs. This functionality is available currently only through vCloud Open API (look for networkProfile). The provider first creates Edge Cluster by specifying resource pool and storage policy pair for primary and secondary Edge Cluster. Then the Edge cluster is assigned to Org VDCs. The Org VDC Edge Gateway nodes are automatically deployed into the Edge Cluster resource pools/datastores.

Legacy Edge deprecation

Org VDC Edge Gateways can no longer be deployed in the legacy mode (without advanced networking enabled). The existing legacy edges must be upgraded to advanced otherwise they are not manageable by vCloud Director. This is usually non-disruptive operation (unless they are still on version 5.5). The upgrade can be performed in bulk (or per org) with cell-management-tool commands:

./cell-management-tool edge-ip-allocation-updates --host <vcd-host> --user administrator --status
./cell-management-tool edge-ip-allocation-updates --host <vcd-host> --user administrator --update-ip-allocations

NSX-T Support

There is no new NSX-T related functionality other than the ability to register NSX-T Manager via UI and support for NSX-T 2.4 (Policy APIs).

SDDC Proxy

This is a completely new feature that allows using vCloud Director as proxy to a dedicated SDDC (vCenter Server with optional NSX). Provider thus can offer multitenant shared services together with dedicated infrastructure with direct access to its management components. vCloud Director becomes Centralized Point of Management (CPOM).

This is quite powerful feature and probably requires its own blog post, but briefly here is how it works.

The service provider deploys dedicated SDDC and register its vCenter Server into vCloud Director that is not going to be used for any Provider VDCs. Then with vCloud OpenAPI creates SDDC object pointing to the vCenter Server and publishes it to an organization. This will create SDDC Proxy which similarly as console proxy securely proxies the tenant all the way to the dedicated vCenter Server (UI/API management endpoints) without the need to expose it to the internet. Additional proxies can be added if needed for additional endpoints (NSX-T Manager, Site Replication Appliance, etc.). The proxying is configured in the users browser by downloading proxy configuration (PAC file) and is protected with time limited access token.

The SDDC appears as another tile in the Datacenter UI.

In order for the tenant to see the new tile type, CPOM extension must be enabled by the provided in the UI plugins (which now can be done from the UI).

vCenter Server published as SDDC is shown in the Provider Admin UI as Tenant published, VC used for PVDC is shown as Service Provider published.

It is possible to override limits, certificate security and ability to use vCenter Server both as SDDC and for VCD Provider VDC with vCloud API /api/admin/extension/settings/cpom.

The SDDC feature also introduces 3 new rights (View SDDC, Manage SDDC and Manage SDDC Proxy).


After the introduction of vCloud Director appliance in the 9.5 release, the new 9.7 appliance now provides not only cell functionality but also embedded PostgreSQL database which can be deployed in active – standby configuration, with synchronous physical replication and semi-automated failover provided by embedded replication manager and manually triggered through appliance ‘promote‘ UI. Usage of an external database with the appliance is no longer supported.

The appliance now uses two vNICs – eth0: Public Network for external traffic (and vCloud Director services such as UI/API, console proxy and internal messaging bus) the other eth1: Private Network for internal traffic – this is the one the embedded DB will use. It is recommended to use two different networks, however both vNICs can be also connected to the same network if that is the expected networking topology, but will need two IPs. Static routes can be configured easily.

Edit (2019/03/29): Correction, the two IPs must be from different subnets due to Photon routing firstboot issue.

It comes in three different sizes:

  • Cell only (no DB): 2 vCPUs, 8 GB RAM
  • Cell with embedded DB small: 2 vCPUs, 12 GB RAM
  • Cell with embedded DB large: 4 vCPUs, 24 GB RAM

Note that there is no DB only node. The cell services are running on all nodes and for high availability three nodes are the recommended minimum. NFS is still hard requirement for any appliance deployment (even with 1 node). Neither Cassandra DB (for VM metrics) nor RabbitMQ (for external integrations) are provided by the appliance and are still need to be optionally deployed.

vCloud Director binaries for non-appliance deployment are still provided but mixing of RHEL/CentOS nodes with appliance nodes is not supported. It is possible (but not that easy) to migrate your existing RHEL environment to appliance based one. The process requires upgrade to version 9.7 first and then migration so environments still using Oracle DB (which is not supported on 9.5 or 9.7) cannot go straight to embedded database and will need deployment of external PostgreSQL DB as intermediate step. Straight upgrade from 9.5 appliance version to 9.7 appliances is not supported and also involves migration.

Installation of certain agents (like vRealize Log Insight) into the appliance is supported as long as they are on the compatibility list but in general the appliance should be considered as a black box unlike RHEL/CentOS cells that can easily run additional software.

Backup of the database is triggered via create-db-backup command from the primary appliance. No automated backup scheduling is available at this time.


  • Microsoft SQL is still supported database for vCloud Director, but marked for future deprecation. PostgreSQL version 9.x is not supported, version 10 is now required.
  • API versions: 32.0, 31.0, 30.0, 29.0 supported, 28.0, 27.0, 26.0, 25.0, 24.0 23.0, 22.0, 21.0, 20.0 supported but marked for deprecation.
  • CloudAPI API Explorer has moved to new location (same as vCloud Director The user must log in to use a provider or tenant specific links:
  • Compatible fully supported Hashicorp blessed Terraform provider 2.1 has been released here accompanied with Golang SDK.
  • pyvcloud Python SDK and vcd-cli have been updated.
  • New vRealize Orchestrator vCloud Director plugin has been released.
  • Scalability and resilience enhancements of VC Proxy (listener) and StatsFeeder (used for VM metric collection). These services are now distributed across all vCloud Director cells (there is no longer 5 minute failover of the listener when the cell running it dies). This is also manifested by multiple VC connection alert emails during start up or VC reconnect action.



vCloud Director Federation with IBM Cloud Identity

IBM Cloud Identity is a cloud SaaS Single Sign-On solution supporting multifactor authentication and identity governance. In this article I will describe how to integrate it with vCloud Director, where vCloud Director acts as a service provider and IBM Cloud Identity as an identity provider.

I have already wrote numerous posts how to federate vCloud Director with Microsoft Active Director Federation Service, VMware Identity Manager and vCenter Single Sign-On.  What makes the integration different for IBM Cloud Identity is that it does not accept vCloud Director metadata XML for simple service provider setup and thus the integration requires more steps.

IBM Cloud Identity is a SaaS service and can be for free set up in a few minutes. It is pretty straight forward and I will skip that part.

As usual, in vCloud Director as Organization Administrator we must prepare the organization for federation. It means making sure that in the in the Administration > Settings > Federation the Entity ID is not empty and up-to-date certificate is generated that will be used to trust and secure the SAML2 assertion exchange between IdP and vCloud Director. The vCloud Director autogenerated self-signed certificate has always 1 year validity, which means once a year it must be regenerated (and the IdP reconfigured). The Organization Administrator is alerted via email when the expiration date is approaching. With vCloud API it is possible to provide your own publicly trusted certificate (with possibly longer validity).

Now we can download the metadata XML from the link provided on the same screen. As mentioned above we unfortunately cannot just upload it to IBM Cloud Identity, instead we need to manually retrieve the correct information from the downloaded spring_saml_metadata.xml file.

We will need the federation certificate (<ds:X509Certificate>) saved as properly formated PEM file:


Assertion consumer service URL which is in my case: https://vcloud.fojta.com/cloud/org/ibm/saml/SSO/alias/vcd

and entityID – in my case IBM.

Now in IBM Cloud Identity we can set up the application:

  1. Upload the vCloud Director federation certificate in Settings  > Certificates > Add Signer Certificate:
  2. Create new application in Applications > Add > Custom Application and set up General details like Description, icon and Application Owners.
  3. Now in Sign-On submenu we can enter all details we have collected from vCloud Director:
    – Sign-on Method: SAML2.0
    – Provider ID: <EntityID>
    – Assertion Consumer Service URL
    – optionally check Use identity provider initiated single sign-on checkbox and provide Target URL in BASE64 encoded string (in my case I used H5 tenant endpoint URL: https://vcloud.fojta.com/tenant/ibm which base64 encoded translates to: aHR0cHM6Ly92Y2xvdWQuZm9qdGEuY29tL3RlbmFudC9pYm0
    – Service Provider SSO URL (same as Assertion Consumer Service URL)
    – check Sign authentication response and pick Signature Althorithm RSA_SHA256
    – check Validate SAML request signature and pick the certificate from step #1
    – optionally check Encrypt assertion
    – Name Identifier: preferred_username
    – NameID Format: urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress

  4. Uncheck: Send all known user attributes in the SAML assertion and instead provide custom list of Attributes to be used. vCloud Director supports the following attributes:
    You can configure the mapping to each. I used preferred_username to be used for vCloud Director username (but alternatively email could be used as well) and did not map Role attribute as I will manage roles in vCloud Director and not leverage Defer to Indentity Provider Role.
  5. Configure Access Policies and Entitlements to specify which users/groups can use vCloud Director.
  6. After saving the application configuration, we can retrieve its SAML2.0 federation metadata from the link provided on the right side of the Sign-On screen.
  7. Back in vCloud Director > Administration > Settings > Federation check Use SAML Identity Provider and upload the downloaded metadata.xml file from the previous step.
  8. Finally we need to import SAML users/groups and assign their role.

If everything done correctly you should be able to login both from IBM Cloud Identity and vCloud Director with the IdP user.

vCenter Server Issue: Recent Tasks Show xxx.label

I had an annoying issue in my lab. Some time ago when I performed vSphere 6.7 PSC convergence my vCenter would stop displaying proper names of tasks in the vSphere Clients UI (both Flex and H5) and show only their placeholders with names like xxx.label.

While there are some KB or communities articles about the issue (and fix) none of them was applicable to my situation (running vCenter Server 6.7U1). I thought that VCSA patches or even deploying new appliance with backup restore would fix it but it did not.

After a little research I found out that the issue is caused by missing catalog.zip file in the /etc/vmware-vpx/locale/ folder. I had another lab with the exactly same vCenter Server build deployed so I just copied that file and transferred it to my vCenter Server Appliance. After service restart via VAMI UI tasks names were back.

I do not know the root cause, but if you have the same issue, give it a go.

NSX-T 2.4: Force Local Account Login

NSX-T supports Role Based Access Control by integrating with VMware Identity Manager which provides access to 3rd party Identity Sources such as LDAP, AD, SAML2, etc.

When NSX-T version 2.3 is integrated with VIDM you would get a choice during the login which type of account you are going to provide (remote or local).

NSX-T version 2.4 no longer provides the option and will always default to the SAML source (VIDM). To force the login with local account provide this specific URL:


PostgreSQL: Beware of the bloat!

During the last weekend my vCloud Director lab died. And the reason was PostgreSQL DB filled up all the disk space. How could that happen in my small lab with one running vApp?

PostgreSQL database when updating rows actually creates new ones and does not immediately delete the (now dead) old rows. That is done in a separate process called vacuuming.

vCloud Director has one pretty busy table named activity_parameters that is continuously updated. And as you can see from the below screenshot (as reported by pgAdmin table statistics) the table size is 26 MB but it is actually taking 24 GB of hard disk space due to the dead rows.

Another quick way to check DB size via psql CLI is:

\c vcloud
SELECT pg_size_pretty (pg_total_relation_size(‘activity_parameters’));

Vacuuming takes times and therefore it can be tuned in postgresql.conf via a few parameters which VMware documents specifically for vCloud Director here or here. Make sure you apply them (I did not). Another issue that could prevent vacuuming to happen is a stale long running transaction on the table.

The fix:

  • short term: add more disk space
  • long term: make sure postgresql.conf is properly configured
    autovacuum = on
    track_counts = on
    autovacuum_max_workers = 3
    autovacuum_naptime = 1min
    autovacuum_vacuum_cost_limit = 2400
  • manually vacuum the activity_parameters table with the following psql CLI command:
    VACUUM VERBOSE ANALYSE activity_parameters;

And do not forget to monitor free disk space on your PostgreSQL host.