vSphere Replication 8.1.1: vcta Issue

VMware has recently released vCloud Availability update that adds vCloud Director compatibility. The tenant needs to install vSphere Replication 8.1.1 which supports vSphere 6.7U1 all the way down to 6.0U3.

The on-prem upgrade from older vSphere Replication appliance (e.g. 6.5.1) is side-by-side. Meaning; you deploy the new 8.1.1 appliance and it connects to the existing one to migrate the data over.

I have noticed that with the new 8.1.1 appliance my cloud replications were not active.

The reason for that was the vcta service was not running on the appliance. The service is responsible for establishing the tunnel with the cloud endpoint and transferring replicated data. Note that the service is not needed for regular vSphere to vSphere replications.

In lab environment where you need to apply a custom endpoint certificate as described here, you might not notice this issue immediately, as the service is started after the certificate change manually with service vcta restart. However, after appliance reboot the service will be down again.

The fix is easy, just enable the service with:

systemctl enable vcta

command from the appliance CLI (via console or SSH if you enabled it before). This is one more thing to remember when setting up cloud replications next to the ESXi vr2c-firewall.vib issue I documented here.


vCloud Director 9.5 Appliance Tips

With vCloud Director 9.5 VMware for the first time released vCloud Director in fully supported appliance format. It is the first iteration of longer process to provide the whole solution in the appliance format, therefore external NFS, database (PostgreSQL/MS SQL) and RabbitMQ is still needed, but this will change in future releases. I would therefore advise today using the 9.5 version only for green field environments and not to mix it with RHEL/CentOS based vCloud Director setups.

If you are going to deploy the appliance here are some tips:

  • Use vSphere Web Client (FLEX) or OVFTool to deploy the appliance. The HTML5 client is not supported.
  • OVF Appliance networking (DNS/Gateway) is provided through Network Profile for the particular port group the appliance is going to be connected to. If it does not exist, vSphere Web Client will create it the first time you deploy appliance to the port group.
  • Appliance is deployed only with one vNIC and one IP address. That means NFS and DB must be accessible from the vNIC (directly or via routed connection). API/UI and Console Proxy are sharing the same IP, but Console Proxy uses port 8443. So you must adjust your Console Proxy Load Balancer network pool to this port.
  • Appliance uses vcloud user with ID 1002 which most likely is different from RHEL/CentOS vcloud user ID and will cause NFS permission issues. That’s why I do not recommend mixed setup.
  • Appliance will copy responses.properties file to the NFS share for other cells to use and connect to the database. Note that the file contains encrypted database login credentials but also the encryption key, so make sure access to NFS share is controlled.
  • If you need to change appliance network configuration after the fact, use the following command: /opt/vmware/share/vami/vami_config_net. The appliance currently has no admin UI.
  • Appliance is Photon based, so you can install additional packages with tdnf install command.

vCloud OpenAPI – Large Payload Issue with Load Balancer

With vCloud Director version 9 new API (cloudapi) based on OpenAPI specification has been introduced next to the legacy based XML API. In vCloud Director 9.5 API Explorer enables consumption of the API directly from the vCloud UI endpoint (read here). Most of the new features are using this OpenAPI such as H5 UI branding, extensions, vRealize Orchestrator service integrations, Cross VDC networking and Roles management.

OpenAPI is very simple to use, JSON based with links provided in headers. However there might be some issues when load balancer with SSL termination is involved as due to the header or payload size the request response will not get through the load balancer.

One such issue is documented in the vCloud Director 9.5 release notes. Attempting to edit Global Rules in the new H5 UI will fail with an error:

unexpected character at line 1 column 1 of the JSON data.

In my case I am using NSX Edge Load balancer with SSL termination and below is the error screenshot:

There are multiple workarounds described in the release notes but actually none worked for me:

  • increasing header maximum at the Edge LB as described in KB 52553 did not help as the number of headers is not the only issue in the particular scenario – the body payload size is as well
  • limiting maximum page size in vCloud Director with cell-management-tool manage-config -n restapi.queryservice.maxPageSize -v 25 fixes the above API call but the subsequent call made by the UI ignores the setting and the response will not get through the LB again.

After some investigations and troubleshooting I discovered that there is a way to increase Edge LB buffer size above the default 32 KB with similar call to the one in the KB 52553:

PUT https://<NSX-Manager>/api/4.0/edges/<Edge-ID>/systemcontrol/config


The above call (NSX 6.4) was enough to fix the issue for me and i can now edit Global Roles in the UI:

vCloud Director 9.5 and VMware Identity Manager Integration

About six months ago I blogged about VMware Identity Manager (VIDM) federation with vCloud Director. That article is still fully valid (and start there if you have not read it yet), however with the introduction of the new tenant HTML 5 user interface I want to describe how you can now chose which UI (legacy or new HTML5) the user will be redirected to.

When a vCloud Director organization is federated with an external IdP there are two different workflows for the login process:

  • In the first workflow the user goes to vCloud Director URL and is redirected to the external IdP to authenticate. After the authentication the user is redirected back to vCloud Director. Now depending on which URL the user initially used, she will be redirected to legacy UI (https://vcloud.example.com/cloud/org/coke) or HTML 5 UI (https://vcloud.example.com/tenant/coke).
  • In the second workflow, the user authenticates to the external IdP first and then is presented with catalog of federated apps and accessible through Single SignOn. Below is an example of VMware Workspace One catalog.

    Clicking a tile with of an app will redirect and sign-in the user directly to the particular app.

The VIDM integration as described in the previous post will however always redirect the user to the legacy UI. So how to force the usage of the new HTML 5 UI?

The is done by adding the Relay State URL to the config of the Web App in VIDM. The tricky part is that (at least as of version 9.5) vCloud Director expects the parameter to be Base64 encoded.

So in my example, the HTML 5 URL for the particular organization I want the user to be redirected to is: https://vcloud.fojta.com/tenant/coke which is Base64 encoded to: aHR0cHM6Ly92Y2xvdWQuZm9qdGEuY29tL3RlbmFudC9jb2tl and that is what must be entered in the Relay State URL field.

I can now create two Web App tiles for the user, so she can choose to which UI to go.

Postman and vCloud Director 9.5 Access Token Authentication

Quick post on how to configure Postman to use the new vCloud API 31.0 bearer token authentication instead of the deprecated authorization token header.

    1. Create your environment if you have not done yet so by clicking the gear icon in the top right corner. Specify environment name and host variable with FQDN to the vCloud Director instance.
    2. Select the environment in the pull down selection box next to the gear icon.
    3. Create new POST request with URL https://{{host}}/api/sessions
      In Headers section add Accept header: application/*+xml;version=31.0
    4. Go to the Tests section and add the following code snippet:
      var bearer = postman.getResponseHeader("X-VMWARE-VCLOUD-ACCESS-TOKEN")

    5. In the Authorization section, select Basic Auth type and provide username (including @org) and password.
    6. Click Send. You should see Status: 200 OK and the response Headers and Body. Save the request into existing or new collection.

      If you did not get 200 OK, fix the error (credentials, or typo).
    7. Notice that in the Headers section of the response is provided the X-VMWARE-VCLOUD-ACCESS-TOKEN. We will not use it for subsequent API calls. It has been picked up and saved into environment variable by the code provided in step #4.
    8. Create new API call. For example: GET https://{{host}}/api/org. Keep the same Accept header. Go to Authorization tab and change the type to Bearer Token and in the token field provide {{X-VMWARE-VCLOUD-ACCESS-TOKEN}}
    9. Click Send. You should get response Status: 200 OK and a list of all Organizations the user is authorized in. Save the new call into collection as Get Organizations.

    Create additional calls into your collection as needed by repeating steps #8-9. You can now reuse your collection anytime also on different environments. Log in first with the POST Login call while specifying correct credentials and then run any other calls from the collection.