Automate Let’s Encrypt Certificates – Part 2

Some time ago I blogged about how I automate acquisition of Let’s Encrypt Certificates for my lab (NSX + vCloud Director) with PowerShell. The old script no longer works due to some changes on Let’s Encrypt side therefore the need for part 2.

To quickly summarize my situation. My lab consists of vCloud Director with multiple cells fronted by NSX-V Load Balancer. I need public certificate for vCloud Director which is uploaded to the NSX-V Load Balancer (that does L7 SSL termination) and to vCloud Director public addresses.

Prerequisites:

  • Web server on the domain your are getting the certificate for. It is necessary for the DNS challenge that proves you own the domain you are requesting the certificate for. I am using IIS on the machine I trigger the script from and supply the root folder where the challenge file needs to be placed.
  • NSX-V API access information – needed to replace the certificate on the NSX-Edge
  • Details about the load balancer (on which Edge it is running and what is the LB application profile of vCloud Director)
  • vCloud Director API access information – needed to upload new certificate and the full chain to vCloud Director public addresses.
  • PowerShell modules: POSH-ACME and PowerCLI

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:admin@fojta.com"
$Domain = "vcloud.fojta.com"
$Vcd = "vcloud.fojta.com"
$VcdAdmin = "administrator"
$VcdPassword = "vcloud"
$IisAcmeRoot = "C:\inetpub\wwwroot\.well-known\acme-challenge"

$RootCert = "-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
"

#Set-PAServer LE_STAGE
Set-PAServer LE_PROD

## Read https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

New-PAAccount -AcceptTOS -Contact $Email
New-PAOrder $Domain

$auths = Get-PAOrder | Get-PAAuthorizations
$token = $auths[0].HTTP01Token
$toPublish = Get-KeyAuthorization $token

## Upload challenge file to the IIS web server
New-Item -Path $IisAcmeRoot -Name $token -Value $toPublish

$auths.HTTP01Url | Send-ChallengeAck
New-PACertificate $Domain
$cert = Get-PACertificate

$IssuerCert = [IO.File]::ReadAllText($cert.ChainFile)
$PrivateKey = [IO.File]::ReadAllText($cert.KeyFile)
$LBCertificate = [IO.File]::ReadAllText($cert.CertFile)

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<trustObject>
<pemEncoding>" + $LBCertificate + $IssuerCert + $RootCert + "</pemEncoding>
<privateKey>" + $PrivateKey + "</privateKey>
<description>vCloud Certificate</description>
</trustObject>"
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Delete Root and intermediate certificate from the Edge as they are not needed
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[0]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[1]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId[2]
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Update vCloud Director with new certificates

$VcdSession = Connect-CIServer $Vcd -User $VcdAdmin -Password $VcdPassword

$Uri = "https://"+$Vcd+"/api/admin/extension/settings/general"
$head = @{"x-vcloud-authorization"=$VcdSession.SessionSecret} + @{"Accept"="application/*;version=33.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$sxml.GeneralSettings.RestApiBaseUriPublicCertChain = $LBCertificate + $IssuerCert + $RootCert
$sxml.GeneralSettings.SystemExternalAddressPublicCertChain = $LBCertificate + $IssuerCert + $RootCert

$r = Invoke-WebRequest -URI $Uri -Method Put -Headers $head -ContentType "application/vnd.vmware.admin.generalSettings+xml" -Body $sxml.OuterXML -ErrorAction:Stop

vCloud Director with TLS-only Connection with External Database

Very brief blog post to document how to install vCloud Director with external database that does not support plain text connections.

In general the process is to do the initial set up with plain text DB connection and then switch to TLS – see the official docs here. That will however not work if the external database supports only TLS connection.

Instead this process must be used:

  1. Import DB certificate (unless it is publicly signed) to cell default JRE keystore.
  2. Use unattended configuration.

Example:

# /opt/vmware/vcloud-director/jre/bin/keytool --import -trustcacerts -keystore /opt/vmware/vcloud-director/jre/lib/security/cacerts -alias psql -file /opt/vmware/vcloud-director/etc/psql.crt

 

Enter keystore password: changeit

 

Owner: CN=10.0.4.64

Issuer: CN=10.0.4.64

Serial number: cb64ae0954184182

Valid from: Fri Nov 22 14:10:39 GMT 2019 until: Sat Nov 21 14:10:39 GMT 2020 Certificate fingerprints:

MD5:  04:4F:8F:C5:9C:CC:D5:E8:F1:50:C1:85:51:D4:FB:AD

SHA1: 22:53:FF:71:A7:EC:9B:D1:74:79:D5:95:46:71:F6:38:A7:E7:F8:4E

SHA256: 08:7C:27:B4:FB:32:04:DE:AF:BB:FE:9D:47:1D:38:46:C8:F4:7C:69:73:DE:8D:CB:BD:2A:A5:B2:11:12:68:DD

Signature algorithm name: SHA256withRSA

Subject Public Key Algorithm: 2048-bit RSA key

Version: 3

 

Extensions:

 

#1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

 

#2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[

CA:true

PathLen:2147483647

]

 

#3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

 

Trust this certificate? [no]:  yes

Certificate was added to keystore

 

 

# /opt/vmware/vcloud-director/bin/configure --unattended -dbhost <DB IP address> -dbname vcloud -dbpassword vcloud -dbtype postgres -dbuser vcloud --database-ssl true –dbport 5423 -ip <cell-ip> --primary-port-http 80 --primary-port-https 443 -cons <cell-ip> --console-proxy-port-https 8443 -k /opt/vmware/vcloud-director/etc/certificates.ks -w <keystore password> -g

.......................................\

Database configuration complete.

 

# /opt/vmware/vcloud-director/bin/cell-management-tool system-setup --email admin@vcloud.com --full-name 'System Admin' --installation-id 33 --password 'VMware1!' -system-name vcd --unattended --user administrator

Creating admin user.

Setting system details.

Completing system setup.

System setup is complete.

vCloud Director Object Storage Extension Reference Design

Just a quick announcement that a vCloud Director Object Storage Extension Reference Design that I wrote is now available at this link.

It deep dives into the use cases, architecture, includes recommended deployment options and description of the new feature of 1.0.1 related to multisite deployments. There are also results of performance tests of the overhead the extension adds over the direct use of the native storage platform.

VCSA Convergence: Failed to Get RPMs

One of my vSphere 6.7 U3 environments I am managing was still using external Platform Services Controller (PSC) from times when it was the prescribed architecture. That is no longer the case so to simplify my management I wanted to get rid of the external PSC via so called Convergence to embedded PSC.

Unfortunately although there is a very nice UI to do this it never worked for me. And I did try multiple times. The error I always ended up was:

Failed to get RPMs.

The /var/log/vmware/converge/converge.log log did not show any error, but what was peculiar there was this entry referring to download of VCSA 6.5.0 files?!

2019-10-29T16:02:01.223Z INFO converge currentURL = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/
2019-10-29T16:02:01.223Z INFO converge Manifest file = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/manifest/manifest-latest.xml

These are obviously not correct for my 6.7 U3 VCSA appliance. This VMware Communities thread finally pushed me in the right direction.

Here are the steps how to resolve this:

  1. Delete content of /root/vema directory on VCSA
  2. Download correct VCSA ISO installation media corresponding to the version of your VC. In my case it was the full 4 GB VMware-VCSA-all-6.7.0-14836122.iso. The patch media VMware-vCenter-Server-Appliance-6.7.0.41000-14836122-patch-FP.iso cca 2 GB large did not work.
  3. Mount the ISO to your VCSA
  4. Re-run the convergence via the UI

vCloud Director H5 UI Error: 431 Request Header Fields Too Large

This is just a short blog post to describe an issue you might get with the tenant or portal HTML UI in vCloud Director where you will see errors in the browser related to request header fields too large.

You will see it more likely with Chrome browser and if your cloud domain is shared with other services. The root cause is that the browser API calls will stop working once the request header gets larger than 8 KBs. While 8 KBs seems like big enough size especially as the request headers vCloud Director uses contain only session ID, JWT token and possibly load balancers headers it also includes all the browser cookies applicable to the vCloud Director domain stored by other web services.

The temporary fix is for the end-user to delete her browser cookies. But is there something the provider could do?

In our case we saw the situation where the vCloud Director instance was on *.vmware.com domain and the browser contained lots of large OAM cookies related to VMware Single Sign-On solution. While those cookies are essential for multiple VMware internal applications, there is no reason for vCloud Director to receive them in every API request. One way how to block the cookies and thus decrease the request header size is to remove them at the load balancer. With NSX-V load balancer this can be accomplished by utilizing SSL L7 termination and an application rule (see my older blog post how to configure NSX-V Edge Load balancer).

In my case the application rule I use is:

Update 2019/10/24: The initial rule would remove all Cookies. I have now amended it with another rule that removes all but vcloud_session_id and vcloud_jwt cookies if they are present.

reqirep ^Cookie:\s.*(vcloud_session_id=[^;]*)|(vcloud_jwt=[^;]*) Cookie:\ \1;\ \2
reqidel ^Cookie:.*OAM*

which deletes all cookies from the request header starting with OAM string