vCloud Director H5 UI Error: 431 Request Header Fields Too Large

This is just a short blog post to describe an issue you might get with the tenant or portal HTML UI in vCloud Director where you will see errors in the browser related to request header fields too large.

You will see it more likely with Chrome browser and if your cloud domain is shared with other services. The root cause is that the browser API calls will stop working once the request header gets larger than 8 KBs. While 8 KBs seems like big enough size especially as the request headers vCloud Director uses contain only session ID, JWT token and possibly load balancers headers it also includes all the browser cookies applicable to the vCloud Director domain stored by other web services.

The temporary fix is for the end-user to delete her browser cookies. But is there something the provider could do?

In our case we saw the situation where the vCloud Director instance was on *.vmware.com domain and the browser contained lots of large OAM cookies related to VMware Single Sign-On solution. While those cookies are essential for multiple VMware internal applications, there is no reason for vCloud Director to receive them in every API request. One way how to block the cookies and thus decrease the request header size is to remove them at the load balancer. With NSX-V load balancer this can be accomplished by utilizing SSL L7 termination and an application rule (see my older blog post how to configure NSX-V Edge Load balancer).

In my case the application rule I use is:

Update 2019/10/24: The initial rule would remove all Cookies. I have now amended it with another rule that removes all but vcloud_session_id and vcloud_jwt cookies if they are present.

reqirep ^Cookie:\s.*(vcloud_session_id=[^;]*)|(vcloud_jwt=[^;]*) Cookie:\ \1;\ \2
reqidel ^Cookie:.*OAM*

which deletes all cookies from the request header starting with OAM string

 

Load Balancing vCloud Director with NSX-T

I just have had a chance for the first time to set up vCloud Director installation that was fronted by NSX-T based load balancer (version 2.4.1). In the past I have blogged how to load balance vCloud Director cells with NSX-V:

Load Balancing vCloud Director Cells with NSX Edge Gateway

vCloud OpenAPI – Large Payload Issue with Load Balancer

NSX-T differs quite a lot from NSX-V therefore the need for this article. The load balancer instance is deployed into the NSX-T Edge Cluster which is a set of virtual or physical NSX-T Edge Nodes. There are also strict sizing guidelines related to the size and number of LB and size of Edge Nodes – see the official docs.

Certificates

Import your VCD public cert in the NSX Manager UI: System > Certificates > Import Certificate. You will need to provide name, full certificate chain, private key and set is as Service Certificate. If it is signed by Enterprise CA you will also before that import the CA cert.

Monitor

Create new monitor in Networking > Load Balancing > Monitors > Add Active Monitor HTTPs

  • protocol HTTPs
  • monitoring port 443
  • default timers
  • HTTP Request Configuration: GET /cloud/server_status, HTTP Request Version: 1
  • HTTP Response Configuration: HTTP response body: Service is up.
  • SSL Configuration: Enabled, Client Certificate: None

Profiles

Application Profile

Networking > Load Balancing > Profiles > Select Profile Type: Application > Add Application Profile > HTTP

Here in the UI we can set only Request Header Size and Request Body Size. Set both to 65535 maximum (65535 for header size and at least 52428800 for body size as ISO/OVA uploads use 50 MB chunks). We will later use API to also configure Response Header Size.

Persistence and SSL Profiles

I will reuse existing default-source-ip-lb-persistence-profile and default-balanced-client-ssl-profile.

Server Pools

Networking > Load Balancing > Server Pools > Add Server Pool

  • Algorithm: Least Connection
  • Active Monitor: picked the one created before
  • Select members: Enter individual members (do not enter port as we will reuse the pool for multiple ports)

 

Virtual Servers

We will add two virtual servers. One for UI/API and another for VM Remote Console connections. For both I have picked the same IP address from the cell logical segment. Ports will be different (443 vs 8443).

vCloud UI

  • Add virtual server: L7 HTTP
  • Ports: 443
  • Ignore Load Balancer placement for now
  • Server Pool: the one we created before
  • Application Profile: the one we created before
  • Persistence: default-source-ip-lb-persistence-profile
  • SSL Configuration: Client SSL: Enabled, Default Certificate: the one we imported before, Client SSL Profile: default-balanced-client-ssl-profile
    Server SSL: Enabled, Client Certificate: None, Server SSL Profile: default-balanced-client-ssl-profile

vCloud Console

  • Add virtual server: L4 TCP
  • Ports: 8443
  • Ignore Load Balancer placement for now
  • Server Pool: the one we created before
  • Application Profile: default-tcp-lb-app-profile
  • Persistence: disabled

Load Balancer

Now we can create load balancer instance and associate the virtual servers with it. Create the LB instance on the Tier 1 Gateway which routes to your VCD cell network. Make sure the Tier 1 Gateway runs on an Edge node with the proper size (see the doc link before).

Networking > Load Balancing > Load Balancers > Add Load Balancer

  • Size: small
  • Tier 1 Gateway
  • Add Virtual Servers: add the 2 virtual servers created in the previous step

Now we have the load balancer up and running you should get all green in the status column. We are not done yet though.

Firstly we need to increase the response header size as vCloud Director Open API sends huge headers with links. Without this, you would get H5 UI errors (Nginx 502 Bad Gateway) and some API calls would fail.  This can be currently done only with NSX Policy API. Fire up Postman or Curl and do GET and then PUT on the following URI:

NSX-manager/policy/api/v1/infra/lb-app-profiles/<profile-name>

in the payload change the response_header_size to at least 10240 50000 bytes.

And finally we will need to set up NAT so our load balanced virtual servers are available both from the outside world (on Tier-0 Gateway) as well from the internal networks. This is quite network topology specific, but do not forget the cells itself must properly connect to the public (load balanced) URL configured in vCloud Director public addresses.

vCloud OpenAPI – Large Payload Issue with Load Balancer

With vCloud Director version 9 new API (cloudapi) based on OpenAPI specification has been introduced next to the legacy based XML API. In vCloud Director 9.5 API Explorer enables consumption of the API directly from the vCloud UI endpoint (read here). Most of the new features are using this OpenAPI such as H5 UI branding, extensions, vRealize Orchestrator service integrations, Cross VDC networking and Roles management.

OpenAPI is very simple to use, JSON based with links provided in headers. However there might be some issues when load balancer with SSL termination is involved as due to the header or payload size the request response will not get through the load balancer.

One such issue is documented in the vCloud Director 9.5 release notes. Attempting to edit Global Rules in the new H5 UI will fail with an error:

unexpected character at line 1 column 1 of the JSON data.

In my case I am using NSX Edge Load balancer with SSL termination and below is the error screenshot:

There are multiple workarounds described in the release notes but actually none worked for me:

  • increasing header maximum at the Edge LB as described in KB 52553 did not help as the number of headers is not the only issue in the particular scenario – the body payload size is as well
  • limiting maximum page size in vCloud Director with cell-management-tool manage-config -n restapi.queryservice.maxPageSize -v 25 fixes the above API call but the subsequent call made by the UI ignores the setting and the response will not get through the LB again.

After some investigations and troubleshooting I discovered that there is a way to increase Edge LB buffer size above the default 32 KB with similar call to the one in the KB 52553:

PUT https://<NSX-Manager>/api/4.0/edges/<Edge-ID>/systemcontrol/config

<systemControl>
    <property>lb.global.tune.http.maxhdr=1024</property>
    <property>lb.global.tune.bufsize=65536</property>
</systemControl>

The above call (NSX 6.4) was enough to fix the issue for me and i can now edit Global Roles in the UI:

How to Enable TLS1.0 on NSX Edge

In one of my previous articles I wrote how NSX upgrade to 6.2.4 impacts PowerCLI as it disables TLS 1.0 ciphers on Edge Load Balancer. The fix for PowerCLI was easy but what if there are other applications still using TLS1.0 that cannot be fixed/updated?

An example is vSphere Replication 6.1.1 which does not support TLS 1.2.

There is workaround. It is possible to create application rule that specifically enables TLS 1.0. The rule syntax is:

tlsv1 enable

application-rule

 

Once the rule is created it can be added in the Advanced Configuration of the virtual Server.

virtual-server

PowerCLI Stops Working After NSX 6.2.4 Upgrade

NSX LBAs of NSX 6.2.3 TLS 1.0 support is deprecated on Edge Service Gateways. So if you are using load balancer with SSL offload, TLS 1.0 ciphers are no longer being supported and those clients that rely on them will not work anymore.

The supported ciphers can be easily checked with nmap. Here is nmap output to website behind NSX Edge 6.2.2 and 6.2.4 load balancer:

NSX 6.2.2 with TLS 1.0
NSX 6.2.2 with TLS 1.0
NSX 6.2.4 without TLS 1.0
NSX 6.2.4 without TLS 1.0

In my case PowerCLI stopped working and could not connect anymore to vCloud Director endpoint behind the Edge load balancer. The error was not very descriptive: The underlying connection was closed: An unexpected error occurred on a send.

PowerCLI Error
PowerCLI Error

Fortunately, it is possible to force PowerCLI to use TLS 1.1/1.2 by editing Windows Registry as described in the KB article: Enabling the TLSv1.1 and TLSv1.2 protocols for PowerCLI (2137109).