Automate Let’s Encrypt Certificates – Part 2

Some time ago I blogged about how I automate acquisition of Let’s Encrypt Certificates for my lab (NSX + vCloud Director) with PowerShell. The old script no longer works due to some changes on Let’s Encrypt side therefore the need for part 2.

To quickly summarize my situation. My lab consists of vCloud Director with multiple cells fronted by NSX-V Load Balancer. I need public certificate for vCloud Director which is uploaded to the NSX-V Load Balancer (that does L7 SSL termination) and to vCloud Director public addresses.

Prerequisites:

  • Web server on the domain your are getting the certificate for. It is necessary for the DNS challenge that proves you own the domain you are requesting the certificate for. I am using IIS on the machine I trigger the script from and supply the root folder where the challenge file needs to be placed.
  • NSX-V API access information – needed to replace the certificate on the NSX-Edge
  • Details about the load balancer (on which Edge it is running and what is the LB application profile of vCloud Director)
  • vCloud Director API access information – needed to upload new certificate and the full chain to vCloud Director public addresses.
  • PowerShell modules: POSH-ACME and PowerCLI

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:admin@fojta.com"
$Domain = "vcloud.fojta.com"
$Vcd = "vcloud.fojta.com"
$VcdAdmin = "administrator"
$VcdPassword = "vcloud"
$IisAcmeRoot = "C:\inetpub\wwwroot\.well-known\acme-challenge"

$RootCert = "-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
"

#Set-PAServer LE_STAGE
Set-PAServer LE_PROD

## Read https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

New-PAAccount -AcceptTOS -Contact $Email
New-PAOrder $Domain

$auths = Get-PAOrder | Get-PAAuthorizations
$token = $auths[0].HTTP01Token
$toPublish = Get-KeyAuthorization $token

## Upload challenge file to the IIS web server
New-Item -Path $IisAcmeRoot -Name $token -Value $toPublish

$auths.HTTP01Url | Send-ChallengeAck
New-PACertificate $Domain
$cert = Get-PACertificate

$IssuerCert = [IO.File]::ReadAllText($cert.ChainFile)
$PrivateKey = [IO.File]::ReadAllText($cert.KeyFile)
$LBCertificate = [IO.File]::ReadAllText($cert.CertFile)

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<trustObject>
<pemEncoding>" + $LBCertificate + $IssuerCert + $RootCert + "</pemEncoding>
<privateKey>" + $PrivateKey + "</privateKey>
<description>vCloud Certificate</description>
</trustObject>"
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Delete Root and intermediate certificate from the Edge as they are not needed
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[0]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[1]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId[2]
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Update vCloud Director with new certificates

$VcdSession = Connect-CIServer $Vcd -User $VcdAdmin -Password $VcdPassword

$Uri = "https://"+$Vcd+"/api/admin/extension/settings/general"
$head = @{"x-vcloud-authorization"=$VcdSession.SessionSecret} + @{"Accept"="application/*;version=33.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$sxml.GeneralSettings.RestApiBaseUriPublicCertChain = $LBCertificate + $IssuerCert + $RootCert
$sxml.GeneralSettings.SystemExternalAddressPublicCertChain = $LBCertificate + $IssuerCert + $RootCert

$r = Invoke-WebRequest -URI $Uri -Method Put -Headers $head -ContentType "application/vnd.vmware.admin.generalSettings+xml" -Body $sxml.OuterXML -ErrorAction:Stop

Which Cloud VMs do not have VM Tools Installed?

Quick PowerCLI one-liner which displays all VMs running in vCloud Director together with their VM Tools version.

Get-CIVM |Select Name, {$_.Org.Name}, {$_.OrgVdc.Name}, {$_.ExtensionData.Section.VMWareTools.Version}

VMTools

Renaming Edge Gateway

Recently one of my customers changed naming convention of vCloud Director Edge Gateways. He renamed them in vCloud Director however the names of virtual machines backing the Edge Gateways stayed the same. Also the names in vShield Manager or NSX Manager did not change. He quickly found out that there is no way to change the name in the GUI of vShield/NSX Manager. This is unfortunate especially for troubleshooting purposes.

Just today I saw the same question internally on our mailing list and the answer is that you can rename the Edge via API. Just GET the particular edge configuration and PUT it back with the changed name. As I recently wrote Powershell scripts that leveraged NSX API I quckly modified it and created two functions – RenameEdgeV5 and RenameEdgeV6.

They both rename the Edge, however only the first one works with vShield Manager. (EDIT 11 June 2015: Unfortunately vShield Manager does not allow renaming of Edges. You will get following error: Edge name cannot be updated. Specified name … does not match … Error code 15013.)

When using NSX Manager you have to use the one that matches the Edge version. The legacy Edges deployed by vCloud Director are version 5. The new NSX Edges are version 6. The API calls to retrieve the Edge info and change it are very similar one uses api 3.0 the other api 4.0.

Here are the scripts:

function RenameEdgeV5 {
<#
.SYNOPSIS Renames vShield Edge
.DESCRIPTION Renames vShield or NSX legacy v5 Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your vShield or NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
.EXAMPLE
PS> RenameEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId -Name newname
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId,
[Parameter(Mandatory=$true)]
[String]$Name
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/3.0/edges"+"/"+$EdgeId
$s = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $s.Content
$sxml.edge.name = $Name
$r = Invoke-WebRequest -Uri $HealthRequest -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

return $r.StatusCode
function RenameEdgeV6 {
<#
.SYNOPSIS Renames NSX Edge
.DESCRIPTION Renames NSX Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
.EXAMPLE
PS> RenameEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId -Name newname
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId,
[Parameter(Mandatory=$true)]
[String]$Name
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/4.0/edges"+"/"+$EdgeId
$s = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $s.Content
$sxml.edge.name = $Name
$r = Invoke-WebRequest -Uri $HealthRequest -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

return $r.StatusCode

} # End of process

} # End of function

Usage example:Usage

vSphere Client recent task activity:
Recent Tasks

 

And NSX Manager Edge Gateway view.RenamedEdges

How to Monitor Health of NSX Edge Gateways

NSX Edge Service Gateways are virtual machines deployed by NSX Manager that provide network services (routing, bridging, load balancing, VPNs, DNS relay, DHCP, …). This makes them quite a critical component in the infrastructure and thus there might be a need to keep a close eye on their availability.

While NSX Manager reports the status of the Edges in the GUI and logs it might take some time to register a change in the health. If you want up-to-date status of an Edge health you need to query the NSX Manager with NSX API. The NSX Manager then retrieves the current status of the Edge. The mechanism of the communication between NSX Manager and Edge appliance(s) depends on the Edge version and the vSphere cluster status:

VIX Communication

This is the legacy mode of communication. NSX Manager utilizes VIX API to query vCenter Server and the ESXi Host which runs the Edge appliance who then via VM Tools talks to the actual VM. This mode of communication is used for the legacy Edges version 5.5.x (deployed via  the compatibility vShield v2 API) and as failback mode when for some reason Message Bus Communication mode is not possible.

VIX Guest Operations
source: https://www.vmware.com/support/developer/vix-api/guestOps50_technote.pdf

 

Message Bus Communication

This is direct (and faster) communication between NSX Manager and the ESXi host (vsfwd process) running the Edge appliance. During the Edge deployment the cluster where the Edge is deployed to must be prepared for NSX and without any issues.

Message Bus
source: NSXvSphereDesignGuidev2.1.pdf

 

To query the Edge health is an expensive operation – it takes time and creates load on NSX Manager. If there is large number of Edges (for example in service provider scenario) this should be considered.

To test the viability of checking at least once in a given time the status of all Edges health I have created simple Powershell function Get-NSXEdgeHealth:

function Get-NSXEdgeHealth {
<#
.SYNOPSIS Gathers Health Status of a NSX Edge
.DESCRIPTION Will query NSX Manager for the health of a particular NSX Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
ID of the Edge to gather health data for.
.EXAMPLE
PS> Get-NSXEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/3.0/edges"+"/"+$EdgeId+"/status"

$h = @{} | select Health, Detail

$r = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$rxml = $r.Content
$h.Health = $rxml.edgeStatus.edgeStatus
$Details = @()
foreach ($Feature in $rxml.edgeStatus.featureStatuses.featureStatus)
{
$n = @{} | select Service, Status
$n.Service = $Feature.service
$n.Status = $Feature.status
$Details += $n
}
$h.Detail = $Details

return ,$h

} # End of process

} # End of function

PowerShell 3.0 or higher and (at least audit) credentials (and connectivity) to NSX Manager are needed.

Usage example:

Example

 

As can be seen the function needs the Edge ID parameter and then returns the overall Edge health and also detailed status of all its network services.

Following health states are defined:

  • green – good. This is the only state that guarantees that the Edge is functional.
  • red – no backing appliance is in service state
  • grey – unknown status (for example undeployed Edge)
  • yellow – intermittent health check failures (if more than 5 consecutive health checks fail the status goes to red)

Following function Get-NSXEdges will collect all Edges in the environment:

function Get-NSXEdges {
<#
.SYNOPSIS Gathers NSX Edges from NSX Manager
.DESCRIPTION Will inventory all of your NSX Edges from NSX Manager
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.EXAMPLE
PS> Get-NSXEdges -NSXManager nsxmgr.fqdn -Username admin -Password password
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password
)

Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
### Connect to NSX Manager via API
$Request = "https://$NSXManager/api/3.0/edges"
$r = Invoke-WebRequest -Uri ($Request+"?startIndex=0&pageSize=1") -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$TotalCount = ([xml]$r).pagedEdgeList.edgePage.pagingInfo.totalCount

$r = Invoke-WebRequest -Uri ($Request+"?startIndex=0&pageSize="+$TotalCount) -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$rxml = $r.Content

### Return the NSX Edges

$Edges = @()

foreach ($EdgeSummary in $rxml.pagedEdgeList.edgePage.edgeSummary)
{
$n = @{} | select Name,Id
$n.Name = $edgeSummary.Name
$n.Id = $edgeSummary.objectId
$Edges += $n
}
return ,$Edges

} # End of process

} # End of function

And here is a sample script leveraging both functions above that continuously displays health status of all the Edges in the environment and also displays the time needed to go through all of them.

$NSXManager = "nsx01.fojta.com"
$Username = "admin"
$Password = "default"

$AllEdges = Get-NSXEdges -NSXManager $NSXManager -Username $Username -Password $Password

DO
{
$StartTime = get-date
foreach ($Edge in $AllEdges)
{
$Health = Get-NSXEdgeHealth -NSXManager $NSXManager -Username $Username -Password $Password -EdgeId $Edge.Id
Write-Host $Edge.Name $Health.Health
}
$Duration = (get-date) - $StartTime
Write-Host
Write-Host "Duration:" $Duration.Minutes "Minutes" $Duration.Seconds "Seconds"
Write-Host
} While ($true)

In my lab it took at least cca 2 seconds to get status of an Edge (depending on the mode of communication and its actual health). It is obvious that most time NSX Manager spends on communication with the ESXi host – so the task should and can be parallelized. While running 5 sessions at the same time the retrieval of health status of one Edge went up to 3-4 seconds (for a green Edge) while the load on NSX Manager went up 14 % (I run NSX Manager only with 2 vCPUs in my lab).

Monitoring

While the article mentions only NSX the scripts should work also with vShield / vCloud Networking and Security Manager.