vCloud Director 9.7 JMS Certificate Issue

Are you still on vCloud Director 9.7 (VCD) in multi-cell configuration? Then you are susceptible to Java Message Service (JMS) certificate expiration issue. Read on.

Background

In multi-cell set up VCD cells need to communicate between themselves. They use shared database but for much faster and efficient communication they also use internal ActiveMQ message bus. It is used for activity sharing and vCenter Server events notifications. If the message bus is dysfunctional it slows any operations almost to halt. For this particular certificate issue you will see in the logs similar message:

Could not accept connection from tcp://<primary-cell-IP:port> : javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

In vCloud Director 9.7 the bus communication become encrypted in preparation for other use cases (read here). On upgrade or new deployment of each cell new certificate was issued by internal VCD_CA with 365 day duration. In vCloud Director 10.0 or VMware Cloud Director 10.1 the certificate is regenerated upon upgrade and its duration is extended to 3 years.

To find out the certificates expiry date, run the following command from any cell:


/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates -status


It will for every cell print out its JMS certificate details:

…

Cell with UUID fd0d2ca0-e357-4aae-9a3b-1c1c5d143d05 and IP 192.168.3.12 has jms certificate: [
[
Version: V3
Subject: CN=vcd-node2.vmware.local
Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

Key: Sun RSA public key, 2048 bits
modulus: 25783371233977292378120920630797680107189459997753528550984552091043966882929535251723319624716621953887829800012972122297123129787471789561707788386606748136996502359020718390547612728910865287660771295203595369999641232667311679842803649012857218342116800466886995068595408721695568494955018577009303299305701023577567636203030094089380853265887935886100721105614590849657516345704143859471289058449674598267919118170969835480316562172830266483561584406559147911791471716017535443255297063175552471807941727578887748405832530327303427058308095740913857590061680873666005329704917078933424596015255346720758701070463
public exponent: 65537
Validity: [From: Wed Jun 12 15:38:11 UTC 2019,
To: Thu Jun 11 15:38:11 UTC 2020]

…

 

Yes, this particular cell’s certificate will expire Jun 12 2020 – in less than two months!

The Fix

Set a calendar reminder and when the certificate expiration day is approaching run the following command.

/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates --certgen

Or upgrade to vCloud Director 10.0 or newer.

Update 21/05/2020: KB 78964 has been published on this topic. Also if the CA signing certificate is expired you will need to disable SSL altogether, restart the cell, regenerate the cert and re-enable SSL.

Automate Let’s Encrypt Certificates – Part 2

Some time ago I blogged about how I automate acquisition of Let’s Encrypt Certificates for my lab (NSX + vCloud Director) with PowerShell. The old script no longer works due to some changes on Let’s Encrypt side therefore the need for part 2.

To quickly summarize my situation. My lab consists of vCloud Director with multiple cells fronted by NSX-V Load Balancer. I need public certificate for vCloud Director which is uploaded to the NSX-V Load Balancer (that does L7 SSL termination) and to vCloud Director public addresses.

Prerequisites:

• Web server on the domain your are getting the certificate for. It is necessary for the DNS challenge that proves you own the domain you are requesting the certificate for. I am using IIS on the machine I trigger the script from and supply the root folder where the challenge file needs to be placed.
• NSX-V API access information – needed to replace the certificate on the NSX-Edge
• Details about the load balancer (on which Edge it is running and what is the LB application profile of vCloud Director)
• vCloud Director API access information – needed to upload new certificate and the full chain to vCloud Director public addresses.
• PowerShell modules: POSH-ACME and PowerCLI

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:admin@fojta.com"
$Domain = "vcloud.fojta.com"
$Vcd = "vcloud.fojta.com"
$VcdAdmin = "administrator"
$VcdPassword = "vcloud"
$IisAcmeRoot = "C:\inetpub\wwwroot\.well-known\acme-challenge"

$RootCert = "-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
"

#Set-PAServer LE_STAGE
Set-PAServer LE_PROD

## Read https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

New-PAAccount -AcceptTOS -Contact $Email
New-PAOrder $Domain

$auths = Get-PAOrder | Get-PAAuthorizations
$token = $auths[0].HTTP01Token
$toPublish = Get-KeyAuthorization $token

## Upload challenge file to the IIS web server
New-Item -Path $IisAcmeRoot -Name $token -Value $toPublish

$auths.HTTP01Url | Send-ChallengeAck
New-PACertificate $Domain
$cert = Get-PACertificate

$IssuerCert = [IO.File]::ReadAllText($cert.ChainFile)
$PrivateKey = [IO.File]::ReadAllText($cert.KeyFile)
$LBCertificate = [IO.File]::ReadAllText($cert.CertFile)

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<trustObject>
<pemEncoding>" + $LBCertificate + $IssuerCert + $RootCert + "</pemEncoding>
<privateKey>" + $PrivateKey + "</privateKey>
<description>vCloud Certificate</description>
</trustObject>"
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Delete Root and intermediate certificate from the Edge as they are not needed
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[0]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[1]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId[2]
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Update vCloud Director with new certificates

$VcdSession = Connect-CIServer $Vcd -User $VcdAdmin -Password $VcdPassword

$Uri = "https://"+$Vcd+"/api/admin/extension/settings/general"
$head = @{"x-vcloud-authorization"=$VcdSession.SessionSecret} + @{"Accept"="application/*;version=33.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$sxml.GeneralSettings.RestApiBaseUriPublicCertChain = $LBCertificate + $IssuerCert + $RootCert
$sxml.GeneralSettings.SystemExternalAddressPublicCertChain = $LBCertificate + $IssuerCert + $RootCert

$r = Invoke-WebRequest -URI $Uri -Method Put -Headers $head -ContentType "application/vnd.vmware.admin.generalSettings+xml" -Body $sxml.OuterXML -ErrorAction:Stop

Automate Let’s Encrypt Certificate for NSX Edge Load Balancer

I needed public certificate for my lab to avoid issues while testing certain libraries that did not allow untrusted connections or importing private Certificate Authority.

Fortunately, there is possibility to issue free public certificates with Let’s Encrypt certificate authority. These certificates are domain validated, which means you need to own the domain for which you issue the certificate. There are three methods how the validation is done but only one can be used in fully automated mode. Why the need for automation? The issued certificates are valid only for 90 days.

To validate the domain ownership you need to publish on publicly accessible web server under the certificate FQDN a specific generated verification string. You do not actually need to publish the service (and the NSX Edge load balancer) to the internet if you do not want to – I just set up a simple webserver with a sole purposed to complete the validation challenge.

So what is the high level process?

1. Own a domain for which you want to have the certificate.
2. Set up publicly accessible web server and point to it a DNS record with the certificate FQDN.
3. Generate challenge string and place it on the web server.
4. Validate the domain and obtain the certificates.
5. Upload the certificates to your NSX Edge Load Balancer.
6. In 60 days repeat from #3.

There are various ways how to automate steps 3-5. I have chosen to do this on Windows with PowerShell but the same could be accomplished on Linux as there are many Let’s Encrypt clients available to chose from.

On a Windows 2012 R2 Server I installed latest Powershell 5, IIS and ACMESharp with PowerShell gallery:

save-module -name ACMESharp
install-module -name ACMESharp

Then I wrote PowerShell script that first goes through the certificate generation and then using NSX API replaces certificate of a specific load balancer.

Note that you need to supply NSX Manager credentials, Edge ID which is running the load balancer, application profile ID which the web server uses (can be easily looked up in NSX UI) and email and domain for the Let’s Encrypt generation process.

Also be aware that Let’s Encrypt has rate limit on how many times a particular certificate can be issued within 7 day period (currently 20).

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:user@example.com"
$Domain = "domain.example.com"


## Generate random alias
$IdentAlias = 'Ident_'+([guid]::NewGuid()).ToString()
$CertAlias = 'Cert_'+([guid]::NewGuid()).ToString()

## Remove and rename old files
If (Test-Path D:\LetsEncrypt\issuer.crt.old) {Remove-Item D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key.old) {Remove-Item D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt.old) {Remove-Item D:\LetsEncrypt\cert.crt.old}

If (Test-Path D:\LetsEncrypt\issuer.crt) {Rename-Item D:\LetsEncrypt\issuer.crt D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key) {Rename-Item D:\LetsEncrypt\cert.key D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt) {Rename-Item D:\LetsEncrypt\cert.crt D:\LetsEncrypt\cert.crt.old}

## Let's Encrypt specific code from https://github.com/ebekker/ACMESharp/wiki/Quick-Start
Import-Module ACMESharp
Initialize-ACMEVault -ErrorAction SilentlyContinue
New-ACMERegistration -Contacts $Email -AcceptTos
New-ACMEIdentifier -Dns $Domain -alias $IdentAlias
Complete-ACMEChallenge $IdentAlias -ChallengeType http-01 -Handler iis -HandlerParameters @{ WebSiteRef = 'Default Web Site' }
Submit-ACMEChallenge $IdentAlias -ChallengeType http-01

$Status = "pending"
Do {
	Start-Sleep -s 5
	$Status = ((Update-ACMEIdentifier $Alias -ChallengeType http-01).Challenges | Where-Object {$_.Type -eq "http-01"}).Status
	}
Until ($Status = "valid")


New-ACMECertificate $IdentAlias -Generate -Alias $CertAlias
Submit-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportCertificatePEM D:\LetsEncrypt\cert.crt
Get-ACMECertificate $CertAlias -ExportKeyPEM D:\LetsEncrypt\cert.key
Update-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportIssuerPEM D:\LetsEncrypt\issuer.crt


$IssuerCert = [IO.File]::ReadAllText("D:\LetsEncrypt\issuer.crt")
$PrivateKey = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.key")
$LBCertificate = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.crt")

## Calculate Issuer Cert Thumbprint
$IssuerCertThumbprint = (Get-PfxCertificate -filepath D:\LetsEncrypt\issuer.crt).Thumbprint.ToLower()

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

## Get all Edge certificates
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/scope/" + $LBEdge
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content

## Find if Issuer Certificate already exists
$exists = $false
foreach ($Certificate in $sxml.certificates.certificate) {
	$Thumbprint = $Certificate.x509Certificate.sha1Hash -replace '[:]'
	if ($Thumbprint -eq $IssuerCertThumbprint) { $exists = $true }
	}

##Upload issuer certificate if it does not exist
if (-Not $exists) {
	$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
	$Body = "
<trustObject>
 <pemEncoding>" +$IssuerCert+ "</pemEncoding>
 <description>Issuer Certificate</description>
</trustObject>"
	$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
	$IssuerId = ([xml]$r).certificates.certificate.objectId
	}
	
##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<trustObject>
 <pemEncoding>" + $LBCertificate + "</pemEncoding>
 <privateKey>" + $PrivateKey + "</privateKey> 
 <description>vCloud Certificate</description>
</trustObject>"
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

How to Generate SSL Certificates for vRealize Operations

vRealize Operations 6 (vRops) has different SSL certificate generation requirements than the older version. I have not found it publicly documented anywhere so here it is:

1. Generate private key:
   openssl genrsa -out vrops.key 2048
2. Create certificate signing request:
   openssl req -new -key vrops.key -out vrops.scr -days 365 -sha256
3. Sign vrops.scr by your Certificate Authority
4. Create PEM text file which contains signed cert from #2, private key from #1 and CA certificate (optionally intermediate certs as well)
5. Go to vRops admin portal and click the certificate icon in the top right corner (next to admin). Install the new certificate by uploading the PEM file from #4.

Wait a little bit and then re-login. Reboot is not necessary.

 

vCloud Usage Meter with Signed SSL Certificates

vCloud Usage Meter is a small virtual appliance used by service providers to measure their VMware product consumption for VSPP (VMware Service Provider Program) type licensing.

I needed to replace the self signed certificate of the web user interface. While there is a KB article 2047572 and also a chapter in the user guide dedicated to the subject neither was correct for my version 3.3.1 installation.

The web interfaces is provided by tc server which stores its certificate keystore in the following location:

/usr/local/tcserver/vfabric-tc-server-standard/um/conf/tcserver.jks

The keystore password is silverpen and the certificate alias is um. The location and password can be changed by editing server.xml in the same directory.

Here is a quick guide how to generate and sign new certificate with java keytool. Note if you need to generate private key externally use the steps described in my older article here.

1. Modify default path to include java keytool location:
   export PATH=$PATH:/usr/java/latest/bin 
2. Go to tc server conf folderd
   cd /usr/local/tcserver/vfabric-tc-server-standard/um/conf/ 
3. Backup current keystore
   mv tcserver.jks tcserver.jks.backup 
4. Generate private key. When asked always use password silverpen
   keytool -genkey -alias um -keyalg RSA -keysize 2048 -keystore tcserver.jks 
5. Modify ownership of the keystore file:
   chown usgmtr tcserver.jks 
6. Create certificate signing request
   keytool -certreq -alias um -keyalg RSA -file vcum.csr -keystore tcserver.jks 
7. Sign CSR with your CA (save certificate as vcum.crt)
8. Import root (and optionally intermediate) certificates if needed
   keytool -import -trustcacerts -alias root -file fojta-dc-CA.cer -keystore tcserver.jks 
9. Import the signed certificate
   keytool -import -alias um -file vcum.crt -keystore tcserver.jks 
10. Verify certificates were successfully imported into keystore
    keytool -list -keystore tcserver.jksKeystore type: JKS

    Keystore provider: SUN
    Your keystore contains 2 entries

    root, Aug 1, 2014, trustedCertEntry,
    Certificate fingerprint (MD5): E3:EE:7F:47:1A:3E:76:07:8F:27:5D:87:54:94:A4:E7
    um, Aug 2, 2014, PrivateKeyEntry,
    Certificate fingerprint (MD5): 26:3C:96:08:63:86:2B:E8:CA:2C:7F:53:6A:B2:EE:FA

11. Restart tc service
    service tomcat restart