Automate Let’s Encrypt Certificate for NSX Edge Load Balancer

NSX LBI needed public certificate for my lab to avoid issues while testing certain libraries that did not allow untrusted connections or importing private Certificate Authority.

Fortunately, there is possibility to issue free public certificates with Let’s Encrypt certificate authority. These certificates are domain validated, which means you need to own the domain for which you issue the certificate. There are three methods how the validation is done but only one can be used in fully automated mode. Why the need for automation? The issued certificates are valid only for 90 days.

To validate the domain ownership you need to publish on publicly accessible web server under the certificate FQDN a specific generated verification string. You do not actually need to publish the service (and the NSX Edge load balancer) to the internet if you do not want to – I just set up a simple webserver with a sole purposed to complete the validation challenge.

So what is the high level process?

  1. Own a domain for which you want to have the certificate.
  2. Set up publicly accessible web server and point to it a DNS record with the certificate FQDN.
  3. Generate challenge string and place it on the web server.
  4. Validate the domain and obtain the certificates.
  5. Upload the certificates to your NSX Edge Load Balancer.
  6. In 60 days repeat from #3.

There are various ways how to automate steps 3-5. I have chosen to do this on Windows with PowerShell but the same could be accomplished on Linux as there are many Let’s Encrypt clients available to chose from.

On a Windows 2012 R2 Server I installed latest Powershell 5, IIS and ACMESharp with PowerShell gallery:

save-module -name ACMESharp
install-module -name ACMESharp

Then I wrote PowerShell script that first goes through the certificate generation and then using NSX API replaces certificate of a specific load balancer.

Note that you need to supply NSX Manager credentials, Edge ID which is running the load balancer, application profile ID which the web server uses (can be easily looked up in NSX UI) and email and domain for the Let’s Encrypt generation process.

Also be aware that Let’s Encrypt has rate limit on how many times a particular certificate can be issued within 7 day period (currently 20).

$Username = "admin"
$Password = "default"
$NSXManager = ""
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = ""
$Domain = ""

## Generate random alias
$IdentAlias = 'Ident_'+([guid]::NewGuid()).ToString()
$CertAlias = 'Cert_'+([guid]::NewGuid()).ToString()

## Remove and rename old files
If (Test-Path D:\LetsEncrypt\issuer.crt.old) {Remove-Item D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key.old) {Remove-Item D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt.old) {Remove-Item D:\LetsEncrypt\cert.crt.old}

If (Test-Path D:\LetsEncrypt\issuer.crt) {Rename-Item D:\LetsEncrypt\issuer.crt D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key) {Rename-Item D:\LetsEncrypt\cert.key D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt) {Rename-Item D:\LetsEncrypt\cert.crt D:\LetsEncrypt\cert.crt.old}

## Let's Encrypt specific code from
Import-Module ACMESharp
Initialize-ACMEVault -ErrorAction SilentlyContinue
New-ACMERegistration -Contacts $Email -AcceptTos
New-ACMEIdentifier -Dns $Domain -alias $IdentAlias
Complete-ACMEChallenge $IdentAlias -ChallengeType http-01 -Handler iis -HandlerParameters @{ WebSiteRef = 'Default Web Site' }
Submit-ACMEChallenge $IdentAlias -ChallengeType http-01

$Status = "pending"
Do {
	Start-Sleep -s 5
	$Status = ((Update-ACMEIdentifier $Alias -ChallengeType http-01).Challenges | Where-Object {$_.Type -eq "http-01"}).Status
Until ($Status = "valid")

New-ACMECertificate $IdentAlias -Generate -Alias $CertAlias
Submit-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportCertificatePEM D:\LetsEncrypt\cert.crt
Get-ACMECertificate $CertAlias -ExportKeyPEM D:\LetsEncrypt\cert.key
Update-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportIssuerPEM D:\LetsEncrypt\issuer.crt

$IssuerCert = [IO.File]::ReadAllText("D:\LetsEncrypt\issuer.crt")
$PrivateKey = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.key")
$LBCertificate = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.crt")

## Calculate Issuer Cert Thumbprint
$IssuerCertThumbprint = (Get-PfxCertificate -filepath D:\LetsEncrypt\issuer.crt).Thumbprint.ToLower()

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

## Get all Edge certificates
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/scope/" + $LBEdge
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content

## Find if Issuer Certificate already exists
$exists = $false
foreach ($Certificate in $sxml.certificates.certificate) {
	$Thumbprint = $Certificate.x509Certificate.sha1Hash -replace '[:]'
	if ($Thumbprint -eq $IssuerCertThumbprint) { $exists = $true }

##Upload issuer certificate if it does not exist
if (-Not $exists) {
	$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
	$Body = "
 <pemEncoding>" +$IssuerCert+ "</pemEncoding>
 <description>Issuer Certificate</description>
	$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
	$IssuerId = ([xml]$r).certificates.certificate.objectId
##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
 <pemEncoding>" + $LBCertificate + "</pemEncoding>
 <privateKey>" + $PrivateKey + "</privateKey> 
 <description>vCloud Certificate</description>
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

How to Generate SSL Certificates for vRealize Operations

vRealize Operations 6 (vRops) has different SSL certificate generation requirements than the older version. I have not found it publicly documented anywhere so here it is:

  1. Generate private key:
    openssl genrsa -out vrops.key 2048
  2. Create certificate signing request:
    openssl req -new -key vrops.key -out vrops.scr -days 365 -sha256
  3. Sign vrops.scr by your Certificate Authority
  4. Create PEM text file which contains signed cert from #2, private key from #1 and CA certificate (optionally intermediate certs as well)
  5. Go to vRops admin portal and click the certificate icon in the top right corner (next to admin). Install the new certificate by uploading the PEM file from #4.

Install CertificateWait a little bit and then re-login. Reboot is not necessary.


vCloud Usage Meter with Signed SSL Certificates

VCUMvCloud Usage Meter is a small virtual appliance used by service providers to measure their VMware product consumption for VSPP (VMware Service Provider Program) type licensing.

I needed to replace the self signed certificate of the web user interface. While there is a KB article 2047572 and also a chapter in the user guide dedicated to the subject neither was correct for my version 3.3.1 installation.

The web interfaces is provided by tc server which stores its certificate keystore in the following location:


The keystore password is silverpen and the certificate alias is um. The location and password can be changed by editing server.xml in the same directory.

Here is a quick guide how to generate and sign new certificate with java keytool. Note if you need to generate private key externally use the steps described in my older article here.

  1. Modify default path to include java keytool location:
    export PATH=$PATH:/usr/java/latest/bin 
  2. Go to tc server conf folderd
    cd /usr/local/tcserver/vfabric-tc-server-standard/um/conf/ 
  3. Backup current keystore
    mv tcserver.jks tcserver.jks.backup 
  4. Generate private key. When asked always use password silverpen
    keytool -genkey -alias um -keyalg RSA -keysize 2048 -keystore tcserver.jks 
  5. Modify ownership of the keystore file:
    chown usgmtr tcserver.jks 
  6. Create certificate signing request
    keytool -certreq -alias um -keyalg RSA -file vcum.csr -keystore tcserver.jks 
  7. Sign CSR with your CA (save certificate as vcum.crt)
  8. Import root (and optionally intermediate) certificates if needed
    keytool -import -trustcacerts -alias root -file fojta-dc-CA.cer -keystore tcserver.jks 
  9. Import the signed certificate
    keytool -import -alias um -file vcum.crt -keystore tcserver.jks 
  10. Verify certificates were successfully imported into keystore
    keytool -list -keystore tcserver.jksKeystore type: JKS

    Keystore provider: SUN
    Your keystore contains 2 entries

    root, Aug 1, 2014, trustedCertEntry,
    Certificate fingerprint (MD5): E3:EE:7F:47:1A:3E:76:07:8F:27:5D:87:54:94:A4:E7
    um, Aug 2, 2014, PrivateKeyEntry,
    Certificate fingerprint (MD5): 26:3C:96:08:63:86:2B:E8:CA:2C:7F:53:6A:B2:EE:FA

  11. Restart tc service
    service tomcat restart


vCAC 6 – How To Generate Signed Certificates

This is the procedure I used to generate and import signed certificates for vCloud Automation Center 6.0.

Identity Appliance

  • Generate private key and certificate signing request with OpenSSL. Common name is FQDN of the Identity Appliance.

openssl.exe req -newkey rsa:2048 -keyout sso.key -nodes -days 3650 -out sso.csr -sha256

Loading ‘screen’ into random state – done
Generating a 2048 bit RSA private key
writing new private key to ‘sso.key’
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
Country Name (2 letter code) [AU]:CZ
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:Prague
Organization Name (eg, company) [Internet Widgits Pty Ltd]
Organizational Unit Name (eg, section) []:vCAC Identity Appliance
Common Name (e.g. server FQDN or YOUR name) []
Email Address []:

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

  • Sign the certificate signing request sso.csr with your CA. Download the signed certificate in Base 64 encoded format (sso.cer).
  • In the SSO > SSL section of Identity Appliance VAMI interface (https://<Identity Appliance FQDN>:5480) choose action: Import PEM encoded Certificate.
  • Paste the private key sso.key to the RSA Private Key field
  • Paste the signed certificate sso.cer to the Certificate Chain section. Append CA root certificate as well.
  • Click Replace Certificate.

Identity Appliance SSL Screen

vCAC Appliance

The process is identical – the only difference is the certificate Common Name and that we are using vCAC Appliance VAMI interface (http://<vCAC Appliance FQDN>:5480 for the import.

IaaS Components

In distributed architecture there can be multiple IaaS components: load balanced website components with Model Manager, Manager service with DEM Orchestrator (active/passive) and multiple Agents and DEM Workers. All those components are Windows based with identical procedure to create domain certificate.

  • Open Microsoft Management Console (mmc.exe) and add Certificates Snap-In (manage Computer account, Local computer).
  • Browse to the Personal Certificates folder and select action Request New Certificate.
  • Request Active Directory Enrollment Policy > Web Server. In the Subject tab configure certificate properties (FullDN, Common Name, Country, etc.), in the General tab type friendly name and in the Private Key tab make private key exportable.
  • Finish by clicking Enroll. Your Domain based CA should now issue the signed certificate.

See my older post that describes this in more detail with screenshots.

vCloud Connector 2.0 and SSL Certificate Replacement

In my previous post about vCloud Connector I went through the various data flows and also explained that it is necessary to have SSL enabled both on the vCC Server and vCC nodes. It is possible to use self-signed certificates or also it is possible to upload via vCC Server/Node VAMI GUI (Virtual Appliance Management Interface) publicly trusted and signed certificates. However when you want to use certificates provided by your own Enterprise Certificate Authority the process is not so straight forward and requires using command line.

The problem statement is following: I have my own Enterprise Certificate Authority which is not publicly trusted and I have issued certificates for my vCC Server and all Nodes without any intermediate certificate. How do I import them?

The vCC VAMI GUI supoprts only import of certificate chain that consist of root>intermediate>certificate. As I do not have intermediate certificate I have to use JAVA keytool command. There are two keystores: cacerts for trusted root certificates and tcserver.jks for the other certificates. My Enterprise CA root certificate must be imported into the former one and the vCC Server/Node certificates to the latter.

Here is the exact procedure:

vCloud Connector Server

1. Log in into the vCC Server console as root and delete the self signed hcserver certificate from tcserver.jks keystore. Note the keystore password (changeme) is hardcoded – do not try to change it.

/usr/java/latest/bin/keytool -delete -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

2. Create new hcserver certificate, where CN (first and last name) is the vCC Server FQDN. Use the same password for the certificate as the keystore password (just hit RETURN when asked).

/usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

What is your first and last name?
What is the name of your organizational unit?
  [Unknown]:vCloud Connector Server
What is the name of your organization?
What is the name of your City or Locality?
  [Unknown]:  Prague
What is the name of your State or Province?
What is the two-letter country code for this unit?
  [Unknown]:  CZ
Is, OU= vCloud Connector Server,, L=Prague, ST=Unknown, C=CZ correct?
  [no]:  yes
 Enter key password for <hcserver>
        (RETURN if same as keystore password):

3. Now we have to create the Certificate Signing Request. We can do it either from command line or from the VAMI GUI (Server>SSL>Generate and download CSR)

/usr/java/latest/bin/keytool -certreq -alias hcserver -file hcserver.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

4. Sign the certificate signing request (hcserver.csr) with your Enterprise CA. I have used the Subordinate Certification Authority certificate template. Upload the signed certificate to the vCC server as file hcserver.cer.

5. Before importing hcserver.cer certificate we need to import the CA root certificate otherwise it would not be able to build trusted chain. Obtain the CA root certificate (in my case named fojta-dc-CA.cer and import to cacerts keystore with following command). The alias must be unique. Note the certificate password (changeit) is hardcoded, do not try to change it (no pun intended).

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Now we can import the hcserver certificate

/usr/java/latest/bin/keytool -import -alias hcserver -file hcserver.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

7. Verify that the import was successful.

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcserver, May 10, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): FC:A3:29:96:0D:CD:E2:04:3D:0F:E9:B6:8B:A0:6F:B8:C8:BF:3E:61

vCloud Connector Node

The process is identical to the vCloud Connector Server certificate replacement with just minor changes. The certificate alias is not hcserver but hcagent and the tcserver.jks keystore location is different. Therefore just briefly:

1. Delete selfsigned hcagent certificate
/usr/java/latest/bin/keytool -delete -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

2. Create new hcagent certificate

/home/admin # /usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

What is your first and last name?
What is the name of your organizational unit?
[Unknown]:vCloud Connector Node
What is the name of your organization?
What is the name of your City or Locality?
[Unknown]:  Prague
What is the name of your State or Province?
What is the two-letter country code for this unit?
[Unknown]:  CZ
Is, OU=vCloud Connector Node,, L=Prague, ST=Unknown, C=CZ correct?
[no]:  yes

Enter key password for <hcagent>
(RETURN if same as keystore password):

3. Create CSR or download from VAMI GUI: Node>SSL>Generate and download CSR

/usr/java/latest/bin/keytool -certreq -alias hcagent -file hcagent.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

4. Sign hcagent.csr with your CA => hcagent.cer

5. Import CA root certificate (fojta-dc-CA.cer)

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Import hcagent certificate

/usr/java/latest/bin/keytool -import -alias hcagent -file hcagent.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

7. Verify

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcagent, May 10, 2013, PrivateKeyEntry
Certificate fingerprint (SHA1): 9E:63:B1:BC:91:FE:7D:84:FC:8C:66:24:B9:1B:B9:73:80:5D:AC:87

Enabling SSL

Now we should be able both on vCC Server and all vCC Nodes to enable SSL (in Server/Node tab, SSL subtab click Enable SSL button). When returning the the SSL tab it seems vCC is not able to get the current SSL status correctly. vCC Server displays an error and vCC Node nothing – not really sure why is this happening, just ignore it.

Enable SSL

Now we should be able in the Nodes tab on the vCC Server edit the Node configuration and uncheck Ignore SSL Certificate. If the certificate replacement was successful, the Node Status shoud be Up.

Ignore SSL Certificate

For more details rather refer to the online vCloud Connector documentation as it contains more information than the PDF docs.