Load Balancing HA vCenter Single Sign-On with NSX

NSX LBOne of the deployment options for vCenter Single Sign-On 5.5 (SSO) is high availability mode. It usually consists of two load balanced SSO nodes deployed in single site configuration. It is quite complex to set up and manage therefore I usually advise customers to avoid such configuration and instead co-deploy SSO together with vCenter Server on the same virtual machine – this results in the same availability of vCenter Service and SSO.

However there are reasons when you cannot do this and need to deploy highly available SSO instance. One is if you want to have multiple vCenter Servers in the same SSO domain with single pane of glass Web Client management. Another is vRealize Automation (vRA) deployment which also requires SSO.

VMware has published two whitepapers about the topic. The first VMware vCenter Server 5.5 Deploying a Centralized VMware vCenter Single Sign-On Server with a Network Load Balancer unfortunately unnecessarily adds even more complexity to the whole process. The paper also describes actve – active load balancing of the nodes which is however unsupported configuration (see here). While active – active load balancing might work with vCenter Server services it does not work with vRealize Automation (vCAC). This is due to the tokens used for solution authentication – WS Trust tokens are stateless but WebSSO are not. Also from what I heard vSphere 6 will not work in active – active configuration at all.

The second whitepaper Using VMware vCenter SSO 5.5 with VMware vCloud Automation Center 6.1 is more recent and while you see vCAC/vRA in its title it still very much applies for pure vSphere environments as well (skip the vRA specific chapters) and it is the one I would recommend. It also describes Active – Passive configuration of F5 Load Balancer.

The topic of this article is however usage of NSX load balancer instead of F5. Contrary to vCNS load balancer, NSX can be configured in Active – Passive mode and thus you can create supported HA SSO configuration with pure VMware solutions.

I will not go too deep in the SSO specific configurations in HA setup (did I mentioned it is complex?) as it is very well described in the second whitepaper mentioned above – instead I will focus on the NSX part of the configurations.

The architecture is like this: two SSO nodes with dedicated NSX load balancer in proxy – on a stick mode. This means LB is not inline of the traffic but instead has only 1 interface and SNAT and DNATs the traffic to the nodes. While inline transparent mode configuration is also possible I believe on a stick config is simpler and provides better resiliency (dedicated LB appliance for each application).

Here are the steps for NSX load balancer configuration:

  1. Deploy Edge Service Gateway for the Load Balancer with one interface preferably in the same subnet as SSO nodes.
  2. Enable Load Balancer feature
    NSX_LB
  3. Upload CA certificate and SSO certificate. See the second whitepaper on how to create SSO certificate.
    Certificates
  4. Configure service monitoring. While you could use the default TCP healh check, I prefer custom HTTPS type healthcheck which is monitoring /lookupservice URL.Service Monitoring
  5. Create Application Profile. During the SSO node configuration before the custom certificates are exchanged on each node you would use simple TCP profile or perhaps SSL passthrough profile (as the SSL certificate configured in NSX would not match self-signed certificate on the nodes). Another alternative is to edit /etc/hosts on each SSO node to fake the VIP hostname to point to the node (this is described in the first white paper). Once you replace the certificates on the nodes you can use SSL termination on the load balancer, configure VIP certificate and Pool Side certificate and also enable Insert X-Forwarded-For HTTP header so in theory we would see from where the authentication request is coming from (unfortunately SSO access log does not display the information). Application Profile
  6. Create Application Rule. Here we will define the logic that will perform the active – passive load balancing. Each SSO node will be in separate pool, with the primary node set up as default. ACL rule is defined to see if the primary node is up. If not we will switch the backend pool to the secondary node. The pool names must match the ones we will create in the next step.

    # detect if pool “SSO_primary” is still UP
    acl SSO_primary_down nbsrv(SSO_primary) eq 0
    # use pool “SSO_secondary” if “SSO_primary” is dead
    use_backend SSO_secondary if SSO_primary_down
    Applicaiton Rule

  7. Create SSO_primary and SSO_secondary pools. Each will have one SSO node with the healthcheck from step 4 and ports 7444. Notice that I have defined the pool member as vCenter VM container object so NSX will retrieve it’s IP address dynamically via VM Tools. While I could hardcode the node IP address this is nice showcase of NSX – vCenter integration. If inline mode you would check the Transparent checkbox for each pool.
    Pool
  8. Now we can create virtual server. We will select Application Profile from step 5, Default Pool from step 7, in the Advanced Tab Application Rule from step 6. For VIP I used the LB default IP (from step 1) and HTTPS 7444 port.
    Virtual Server
  9. As a last step do not forget to disable firewall or create firewall rule for the IP and port define in the previous step.

vCloud Director – vCenter Single Sign-On Integration Troubleshooting

The access to vCloud Director provider context (system administrator accounts) can be federated with vCenter Single Sign-On. This means that when vCloud system administrator wants to log into the vCloud Director (http://vCloud-FQDN/cloud) he is redirected to vSphere Web Client where he needs to authenticate and then redirected back to vCloud Director.

VCD_SSO_Federation

Here follows collection of topics that might be useful when dealing when troubleshooting the federation:

1. When the SSO federation is enabled the end-user is always redirected to vSphere Web client. If you want to use local authentication use http://vCloud-FQDN/cloud/login.jsp and type local or LDAP account (if LDAP is configured).

2. If you enabled SSO federation and the vCenter SSO is no longer available, you cannot unregister its lookup service. To do this go to vCloud database and in dbo.config table clear the lookupservice.url value.

3. In case you are using self-signed untrusted certificate on the vSphere web client some browsers (Firefox) might not display the Add Security Exception button when being redirected. Therefore open first the vSphere web client page directly, create the security exception and then the redirection from vCloud website should work.

4. HTTP ERROR 500. Problem accessing /cloud/saml/HoKSSO/alias/vcd. Reason: Error determining metadata contracts
….
Metadata for issuer https://xxx:7444/STS wasn’t found

This error might appear after the vSphere web client SSO authentication when the user is redirected back to vCloud portal. To understand this error let’s first talk about what is going on in the background. When the SSO federation is enabled, vCloud Director establishes trust with vCenter SSO. The trust is needed so the identity provider (SSO) knows that the request for authentication is not malicious (phishing) and also the service provider (vCloud Director) needs to be sure the reply comes from the right identity provider.

The trust is established when the federation is configured with metadata exchange that contains keys and some information about the other party. The SSO metadata can be seen in vCloud Director database in the dbo.saml_id_provider_settings table. Now during the actual authentication process if for some reason the security token reply comes from different source than the one expected based on the identity provider metadata, you will get this error.

This issue might happen for example if the vCenter SSO hostname has been changed. In this particular case which I encountered it happened on vCenter Server Appliance 5.1. The SSO has been initiated before the hostname was set. So the identity provider response came with metadata containing the issuer’s IP address instead FQDN which the service provider (VCD) expected based on the SSO endpoint address. The issuer information did not get updated after the hostname change.

The VCSA 5.1 the issuer information is stored in SSO PostgreSQL DB. These are the steps to change it:

  1. Get the SSO DB instance name, user and password

cat /usr/lib/vmware-sso/webapps/lookupservice/WEB-INF/classes/config.properties

db.type=postgres
db.url=jdbc:postgresql://localhost:5432/ssodb
db.user=ssod
db.pass=rTjsz9PRPLOFswBkJTCNAAVp
`

  1. Connect to the DB with the password retrieved from the previous step (db.pass=…)

/opt/vmware/vpostgres/1.0/bin/psql ssodb -U ssod

Retrieve the STS issuer:

ssodb=> select issuer from ims_sts_config;

If the issuer is really incorrect update it with following command:

ssodb=> update ims_sts_config SET issuer='https://FQDN:7444/STS'

Note: I need to credit William Lam for helping me where to find the SSO DB password.

vCloud Director and Single-Sign-On (SAML)

In December last year i wrote a blog post about vCloud Director and SSPI Authentication. In the post I stated that besides using SSPI – which is Microsoft proprietary interface on top of Active Directory, the tenants can use Security Assertion Markup Language (SAML) standard to integrate with their identity provider. VMware has tested SAML2 integration with OpenAM (described in detail in vCloud Architecture Toolkit Implementation Examples) and Active Directory Federation Services (ADFS). However just recently there appeared another supported identity provider – our own VMware Horizon Workspace. The following whitepaper describes the integration in detail: Using VMware Horizon Workspace to Enable SSO in VMware vCloud Director 5.1.

In this post I will provide short step-by-step description of all the necessary steps that you as the vCloud Organization Administrator must take. The assumption is that you have on premise Horizon Workspace integrated with company Active Directory and want to use it for connecting private or public vCloud Director organizations.

  1. Download Horizon Identity provider metadata XML file from: https://<horizon_workspace_URL>/SAAS/API/1.0/GET/metadata/idp.xml
  2. In the target cloud go to Administration > Settings > Federation menu and check Use SAML Identity Provider and upload the idp.xml file
  3. Still on the same page regenerate the certificate and click apply
  4. Download the certificate from the url: https://<vcloud_URL>/cloud/org/<orgname>/saml/metadata/alias/vcd
  5. Log out from the cloud
  6. Log back in, you will need to change the URL to go directly to the local authentication: https://<vcloud_URL>/cloud/org/<orgname>/login.jsp
  7. In the Administration > Members > Users (or Groups) import Users (or Groups) by clicking the icon with arrow. Change the Source to SAML and type the user names or group names.
  8. Back in Horizon Workspace admin interface create a new Web Application in the catalog
  9. Fill in the following data:
    • Authentication Profile: SAML 2.0 POST profile
    • Login Redirection URL: https://<vcloud_URL>/cloud/org/<orgname>/
    • Check: Include Destination
    • Check: Sign Response
    • Check: Sign the Assertion
    • Configure via Metadata XML
    • Paste the certificate from point 4 into the Meta-data XML box
    • Add Attribute Mapping as seen in the screenshot
      Attribute Mapping
    • Save the page
  10. Edit the newly created Web Application and assign Entitlements (either specific users or a group). These should be the same users as in step 7.
  11. Now log into the Horizon as the entitled user and click the application icon. You should now get direct access into the vCloud Director.

Horizon Workspace

 

Edit 2 July 2013: In order to get SAML Groups working following is needed.

In step 9 create also group mapping. The group name must be hardcoded, but that should not be such a problem as a different web application in Horizon Workspace can be created for each group/role mapping. I have created hardcoded mapping to group name OrgAdministrators.

Attribute Mapping with Group

 

Then in step 7 the group can be imported and the correct role assigned.

SAML Group Import

vCloud Director and SSPI Authentication

vCloud Director 5.1 brings Single Sign On (SSO) functionality which simplifies the way users log in to the vCloud Director portal. There is some confusion with vCenter SSO so let me first state all the SSO options:

  • When vCloud Director is integrated with vSphere Lookup Service (System > Administration > System Settings > Federation menu) then vCloud Administrators can use vCenter SSO for authentication. The login screen is replaced with vSphere Client login screen.
  • The tenants can self configure SAML identity provider to enable SSO for their user access to the portal. This can be done by Organization Administrator in the Administration > Settings > Federation menu. Currently supported providers are OpenAM and Active Directory Federations Services (ADFS). This option is ideal for Public Clouds when there is no connectivity between vCloud Director cells and customers LDAP servers.
  • The vCloud Administrator can enable Microsoft proprietary Security Support Provider Interface (SSPI) to simplify authentication with Active Directory for the organization users.

The third option is not documented in any way so i decided to research it and write down how to get it working.

Following diagram shows the authentication process:

SSPI

Although in the diagram vCloud Director (VCD) does not talk directly to Active Directory server it is still necessary there is direct connectivity between the cells and AD servers, because group membership information is retrieved from AD by the cells. The client must be logged into the domain and use browser that enables integrated authentication. The client also has to use the same DNS server as the active directory. This implies that the SSPI authentication is valid only for private cloud use case.

Here are the steps to get it working:

  1. For given organization enable custom LDAP with kerberos authentication. Here is a nice detailed article how to do it.
  2. In AD create a group for the users and import it to vCloud Director and assign a vCloud Director role.
  3. Again on the AD server in Active Directory Users and Computer create a computer account for vCloud Director. I created account VCLOUD under Computers.
  4. Map this account to the Security Principal Name (SPI) which has following form: HTTP/<vCloud FQDN>@AD REALM. This can be done with setspn command from the AD server:

    setspn -a HTTP/vcloud.fojta.com VCLOUD$

    (note the $ character at the end of the Computer account)

  5. Now we need to export the encryption key used by the Kerberos token issuer to encrypt the token and upload it to vCloud Director. This can be done from command line (again on the AD server) with ktpass utility:

    ktpass -princ HTTP/<VCD FQDN>@AD REALM -pass * -mapuser <computer account>$@domain -out HTTP.keytab -ptype KRB_NT_PRINCIPAL /crypto RC4-HMAC-NT
     

    in my case I used:
    ktpass -princ HTTP/vcloud.fojta.com@FOJTA.COM -pass * -mapuser VCLOUD$@FOJTA.COM -out VCLOUD.HTTP.keytab -ptype KRB5_NT_PRINCIPAL /crypto RC4-HMAC-NT

    type any password and confirm the reset.

    ktpass

  6. Now we need to take the created keytab file (VCLOUD.HTTP.keytab) and use it for the SSPI configuration. In the vCloud Director organization Custom LDAP configuration go to the SSPI section, enable it, enter the Service Principal Name from step #4 and upload the keytab file from step 5#.
    LDAP
  7. To get the single sign on experience some settings must be done in the client web browser:
    • Internet Explorer: go to Internet Options, Security, Local intranet, Sites and add vCloud Director FQDN to the zone trusted sites. Then edit the security level for this zone by clicking Custom Level button and at the bottom of the list in User Authentication Logon section enable Automatic logon only in Intranet zone.
    • Firefox: open about:config URL and search for network.negotiate and add vCloud Director FQDN to the network.negotiate-auth.trusted-uris.