vCloud OpenAPI – Large Payload Issue with Load Balancer

With vCloud Director version 9 new API (cloudapi) based on OpenAPI specification has been introduced next to the legacy based XML API. In vCloud Director 9.5 API Explorer enables consumption of the API directly from the vCloud UI endpoint (read here). Most of the new features are using this OpenAPI such as H5 UI branding, extensions, vRealize Orchestrator service integrations, Cross VDC networking and Roles management.

OpenAPI is very simple to use, JSON based with links provided in headers. However there might be some issues when load balancer with SSL termination is involved as due to the header or payload size the request response will not get through the load balancer.

One such issue is documented in the vCloud Director 9.5 release notes. Attempting to edit Global Rules in the new H5 UI will fail with an error:

unexpected character at line 1 column 1 of the JSON data.

In my case I am using NSX Edge Load balancer with SSL termination and below is the error screenshot:

There are multiple workarounds described in the release notes but actually none worked for me:

  • increasing header maximum at the Edge LB as described in KB 52553 did not help as the number of headers is not the only issue in the particular scenario – the body payload size is as well
  • limiting maximum page size in vCloud Director with cell-management-tool manage-config -n restapi.queryservice.maxPageSize -v 25 fixes the above API call but the subsequent call made by the UI ignores the setting and the response will not get through the LB again.

After some investigations and troubleshooting I discovered that there is a way to increase Edge LB buffer size above the default 32 KB with similar call to the one in the KB 52553:

PUT https://<NSX-Manager>/api/4.0/edges/<Edge-ID>/systemcontrol/config

<systemControl>
    <property>lb.global.tune.http.maxhdr=1024</property>
    <property>lb.global.tune.bufsize=65536</property>
</systemControl>

The above call (NSX 6.4) was enough to fix the issue for me and i can now edit Global Roles in the UI:

Advertisements

vCloud Director 9.5 and VMware Identity Manager Integration

About six months ago I blogged about VMware Identity Manager (VIDM) federation with vCloud Director. That article is still fully valid (and start there if you have not read it yet), however with the introduction of the new tenant HTML 5 user interface I want to describe how you can now chose which UI (legacy or new HTML5) the user will be redirected to.

When a vCloud Director organization is federated with an external IdP there are two different workflows for the login process:

  • In the first workflow the user goes to vCloud Director URL and is redirected to the external IdP to authenticate. After the authentication the user is redirected back to vCloud Director. Now depending on which URL the user initially used, she will be redirected to legacy UI (https://vcloud.example.com/cloud/org/coke) or HTML 5 UI (https://vcloud.example.com/tenant/coke).
  • In the second workflow, the user authenticates to the external IdP first and then is presented with catalog of federated apps and accessible through Single SignOn. Below is an example of VMware Workspace One catalog.

    Clicking a tile with of an app will redirect and sign-in the user directly to the particular app.

The VIDM integration as described in the previous post will however always redirect the user to the legacy UI. So how to force the usage of the new HTML 5 UI?

The is done by adding the Relay State URL to the config of the Web App in VIDM. The tricky part is that (at least as of version 9.5) vCloud Director expects the parameter to be Base64 encoded.

So in my example, the HTML 5 URL for the particular organization I want the user to be redirected to is: https://vcloud.fojta.com/tenant/coke which is Base64 encoded to: aHR0cHM6Ly92Y2xvdWQuZm9qdGEuY29tL3RlbmFudC9jb2tl and that is what must be entered in the Relay State URL field.

I can now create two Web App tiles for the user, so she can choose to which UI to go.

Postman and vCloud Director 9.5 Access Token Authentication

Quick post on how to configure Postman to use the new vCloud API 31.0 bearer token authentication instead of the deprecated authorization token header.

    1. Create your environment if you have not done yet so by clicking the gear icon in the top right corner. Specify environment name and host variable with FQDN to the vCloud Director instance.
    2. Select the environment in the pull down selection box next to the gear icon.
    3. Create new POST request with URL https://{{host}}/api/sessions
      In Headers section add Accept header: application/*+xml;version=31.0
    4. Go to the Tests section and add the following code snippet:
      var bearer = postman.getResponseHeader("X-VMWARE-VCLOUD-ACCESS-TOKEN")
      pm.environment.set("X-VMWARE-VCLOUD-ACCESS-TOKEN",bearer)
      

    5. In the Authorization section, select Basic Auth type and provide username (including @org) and password.
    6. Click Send. You should see Status: 200 OK and the response Headers and Body. Save the request into existing or new collection.

      If you did not get 200 OK, fix the error (credentials, or typo).
    7. Notice that in the Headers section of the response is provided the X-VMWARE-VCLOUD-ACCESS-TOKEN. We will not use it for subsequent API calls. It has been picked up and saved into environment variable by the code provided in step #4.
    8. Create new API call. For example: GET https://{{host}}/api/org. Keep the same Accept header. Go to Authorization tab and change the type to Bearer Token and in the token field provide {{X-VMWARE-VCLOUD-ACCESS-TOKEN}}
    9. Click Send. You should get response Status: 200 OK and a list of all Organizations the user is authorized in. Save the new call into collection as Get Organizations.

    Create additional calls into your collection as needed by repeating steps #8-9. You can now reuse your collection anytime also on different environments. Log in first with the POST Login call while specifying correct credentials and then run any other calls from the collection.

vCloud Director – vCenter Server Relationship

vCloud Director as a cloud management platform needs resources to provision the target workloads to. These resources are provided by vCenter Server (compute, storage, networks) and NSX Manager (networks and networking services).

In the past vCloud Director required tight grip on those resources, so the recommendation and best practice was to dedicate them to the particular vCloud Director instance. System admins were discouraged to run additional workloads not managed by vCloud Director on them. However that has changed recently, therefore the need for this blog article.

vCenter Server Extension

vCloud Director used to register itself as a vCenter Server extension. That allowed to ‘protect’ VCD managed VMs with a special icon and a warning pop up during vCenter edits of such VMs.

vCloud Director specific VM icon

Today, vCloud Director is quite resilient against changes on a particular VM done directly in vCenter Server, so there is no more need for those warnings. vCloud Director 9.5 thus no longer register itself as vCenter Server extension, so you will no longer see these icons and pop up warnings.

As a side note you will also see a change in the VM naming. The long UUID is no longer added to the VM name and is replaced by shorter 4 digit random characters.

Host Preparation

In the past during creation of a Provider VDC, the system admin was asked for ESXi host credentials. This was needed to upload cloud agent vib that was used for certain features (thumbnails, VCDNI network encapsulation). All these features were either replaced by different mechanism or deprecated, so there is no need to upload any vcloud vibs to ESXi hosts anymore.

Additionally, custom attribute system.service… used to be set for each vCloud Director managed host and vCloud Director managed VM. This provided a way to control where vCenter DRS could vMotion VMs through this host to VM compatibility option. Disabling a host would remove the custom property. vCloud Director VM could not be vMotioned to unprepared host as vCenter would scream the host is incompatible with the VM.

 

Host Custom Attributes
VM Custom Attribute

vMotion to Unprepared Host Error

In vCloud Director 9.5 this mechanism was completely eliminated. When host is put into maintenance mode, it is considered unavailable for vCloud Director, therefore there is no need anymore to disable it first in vCloud Director. You will  no longer see any host preparation dialog and the hosts section is simplified to bare minimum.

So What About the Relationship?

As you can see the vCloud Director – vCenter Server relationship is very loose. In fact it is no longer monogamous, meaning vCenter Server can be married (associated) with multiple vCloud Director instances at the same time.

Why would you do that?

I can think of three use cases, but obviously our smart service providers will come up with more.

Use case 1: Test & Dev

Do you need to test new vCloud Director release? Or provide test instance of vCloud Director for your internal developers? No need to spin up whole vSphere + NSX environment with storage, etc. Just deploy one VM with vCloud Director bits (you can even use the appliance if you have external DB and NFS ready) and point it to the existing vSphere/NSX endpoints.

Use case 2: Whitelabeling / Reselling

To enable three tier mode, where SP provides infrastructure, reseller will get its own vCloud Director instance (branded) to resell to their end customers. SP needs to setup one big vSphere/NSX infrastructure and have an automated way to deploy VCD instances on top of it. The reseller gets its own instance with system admin equivalent rights and manages its own tenants.

Use case 3: Uber Org Admin Role

Some end-customers request bigger than Org Admin role. They want to create their own organizations and Org VDCs to better align with their business groups. SP can dedicate whole VCD instance to such customer. Without the need of provisioning dedicated vSphere/NSX as well.

Any Caveats, Recommendations?

  • Segment vSphere environment with Clusters and Resource Pools for each VCD instance.
  • Use different VCD instance names and IDs
  • Use separate accounts both for vCenter Server and NSX for each VCD instance. Give each account permission only to resources it should see (use vCenter No-Access privilege on clusters/RP/folders it should not see)
  • Dedicate storage resources to each VCD
  • Use separate NSX transport zone for each VCD instance
  • Monitor the load of multiple VCD listeners on the single VC. Scale out VCs if needed. VMware does not test this kind of setup at scale.

Fun Fact

You can in fact vMotion running VM from one vCloud Director environment to another one. To do so, you will move it in vCenter from the source Org VDC RP to the destination Org VDC RP. You must also move it out of the VM Folder (remember the No Access privilege?) to be visible to the target VCD. Obviously it needs to be connected to the right target networks.

Finally, you will need to remove the original vCloud UUID (with PowerCLI or similar) and let it be auto-discovered by the target VCD. There is no auto-removal from the original VCD, so you will need to use the process described here.

 

 

 

How to Change vCloud Director Installation ID

During installation of vCloud Director you must supply installation ID from range 1-63. Its purpose is very similar to vCenter Server ID, it is used for generation of unique MAC addresses for VMs running in the vCloud Director instance. The MAC address format is 00:50:56:ID:xx:xx. vCloud Director overrides MAC vCenter Server assignments.

Obviously on the same layer 2 network MAC addresses must always be unique so when using L2 VPN or similar network extension mechanisms a care should be given that each vCloud Director instance participating in such network extension has a different installation ID to guarantee non-overlapping MAC addresses of deployed VMs.

Until vCloud Director 9.5 it was not possible to change its installation ID. The reason was, that during the installation vCloud Director actually generates all possible MAC addresses in its database, so that table would have to be regenerated with the ID change.

This can now be accomplished with cell-management-tool mac-address-management CLI command that takes care of the MAC address table regeneration and also informs how many MAC addresses still exist that are based on the old ID. Those existing VMs will keep its old MAC unless it is manually reset/regenerated from vCloud Director UI or via vCloud API.

The CMT command can either regenerate MAC addresses with a specific ID that can differ from the installation ID (option –regenerate-with-seed), or you can change the installation ID in the database first (GSS alert!) and just use –regenerate option.

The pgAdmin screenshot below shows the ID location in the vCloud DB. For production setups this should be done with help from GSS.

Finally, here is a screenshot showing the –show-seed option listing the actual MAC usage based on the seed IDs.