How to Enable TLS1.0 on NSX Edge

In one of my previous articles I wrote how NSX upgrade to 6.2.4 impacts PowerCLI as it disables TLS 1.0 ciphers on Edge Load Balancer. The fix for PowerCLI was easy but what if there are other applications still using TLS1.0 that cannot be fixed/updated?

An example is vSphere Replication 6.1.1 which does not support TLS 1.2.

There is workaround. It is possible to create application rule that specifically enables TLS 1.0. The rule syntax is:

tlsv1 enable



Once the rule is created it can be added in the Advanced Configuration of the virtual Server.


Collect vCloud Director Cell Logs with Log Insight Agent

vcenter-log-insight-logoWhile it is possible to redirect vCloud Director cell logs by editing file to remote syslog server (see KB 2004564) there is an alternative agent based method utilizing vRealize Log Insight.

Log Insight agent is installed on each cell and then remotely managed from Log Insight server. Here are some advantages of this approach:

  • no manual edits of log4j file which gets overwritten with each upgrade
  • as we do not rely log4j logger we are able to collect also API request log files which are generated by Jetty
  • agent uses reliable TCP communication as opposed to unreliable UDP
  • we no longer rely on source IP to identify sender; cells can use source NAT (with single IP) to communicate with Log Insight server and we can still distinguish them
  • we can remotely change which logs we want to monitor (info vs debug)
  • and much more

Here is quick configuration how to:

  1. Download Log Insight Agent from Log Insight Server. It is already customized installation for your vRLI server. Administration > Agents > scroll down > Download Log Insight Agent Version 3.6.0 > pick rpm package
  2. Upload rpm file to each cell and install it with rpm -i VMware-Log-Insight-Agent-3.6.0-4148343.noarch_XXX.rpm
  3. Back in Agents configuration create active agent group from vCloud Director Cell Server template (copy template icon)
  4. Create hostname filter (use ? for any character substitution, you can add multiple entries in one line for ‘logical or’ or multiple lines for ‘logical and’
  5. Optionally edit agent configuration to include additional files or directories


PowerCLI Stops Working After NSX 6.2.4 Upgrade

NSX LBAs of NSX 6.2.3 TLS 1.0 support is deprecated on Edge Service Gateways. So if you are using load balancer with SSL offload, TLS 1.0 ciphers are no longer being supported and those clients that rely on them will not work anymore.

The supported ciphers can be easily checked with nmap. Here is nmap output to website behind NSX Edge 6.2.2 and 6.2.4 load balancer:

NSX 6.2.2 with TLS 1.0

NSX 6.2.2 with TLS 1.0

NSX 6.2.4 without TLS 1.0

NSX 6.2.4 without TLS 1.0

In my case PowerCLI stopped working and could not connect anymore to vCloud Director endpoint behind the Edge load balancer. The error was not very descriptive: The underlying connection was closed: An unexpected error occurred on a send.

PowerCLI Error

PowerCLI Error

Fortunately, it is possible to force PowerCLI to use TLS 1.1/1.2 by editing Windows Registry as described in the KB article: Enabling the TLSv1.1 and TLSv1.2 protocols for PowerCLI (2137109).




Gathering Health Status of vCloud Director Edge Gateways

Some time ago I wrote about how to monitor health of NSX Edge Gateways. In this blog post I will show how to get health and other info about vCloud Director Edge Gateways with PowerCLI.

PowerCLI already includes vCloud Director related cmdlets, unforunatelly there is none related to Edge Gateways. This can be easily remediated by using vCloud API however to get detailed information about Edge health we must use NSX API. As of vCloud Director 8.0 the service provider can easily get NSX Edge ID which is backing up particular vCloud Director Edge as a new type GatewayBacking was added.

What follows is an example of function that collects as much information as possible (interfaces, network services, size, syslog, default gateway, health of all services, Org, Org VDC and Provider VDC) about all Edge Gateways from PowerCLI, vCloud API and NSX API.

Note: there is dependency on the Get-NSXEdgeHealth function.

function Get-CIEdgeGateways {
<# .SYNOPSIS Gathers Edge Gateways from vCloud Director and all info through PowerCLI, vCloud API and NSX API .DESCRIPTION Will inventory all of your vCloud Director Edge Gateways .NOTES Author: Tomas Fojta #>
	[String]$NSXUsername = "admin",

	$output = @();
	$EdgeGWs = Search-Cloud -QueryType EdgeGateway

	Foreach ($Edge in $EdgeGWs) {
		$Edgeview = $Edge | get-ciview
		$Vdc = get-OrgVdc -Id ($Edge.PropertyList.Vdc) -ErrorAction SilentlyContinue
		$webclient = New-Object
		$webclient.Headers.Add("accept",$EdgeView.Type + ";version=9.0")
		[xml]$EGWConfXML = $webclient.DownloadString($EdgeView.href)
		$n = "" | Select Name,Description,EdgeBacking,Interfaces,Firewall,NAT,LoadBalancer,DHCP,VPN,Routing,Syslog,Size,HA,DNSRelay,DefaultGateway,AdvancedNetworking, Org, TenantId, OrgVDC, OrgVDCId, ProviderVDC, ProviderVDCId, Health
		$n.Name = $EGWConfXML.EdgeGateway.Name
		$n.Description = $EGWConfXML.EdgeGateway.Description
		$n.EdgeBacking = $EGWConfXML.EdgeGateway.GatewayBackingRef.gatewayId
		$n.Interfaces = $EGWConfXML.EdgeGateway.Configuration.GatewayInterfaces.GatewayInterface
		$n.Firewall = $EGWConfXML.EdgeGateway.Configuration.EdgegatewayServiceConfiguration.FirewallService.FirewallRule
		$n.NAT = $EGWConfXML.EdgeGateway.Configuration.EdgegatewayServiceConfiguration.NatService.NatRule
		$n.LoadBalancer = $EGWConfXML.EdgeGateway.Configuration.EdgegatewayServiceConfiguration.LoadBalancerService.VirtualServer	
		$n.DHCP = $EGWConfXML.EdgeGateway.Configuration.EdgegatewayServiceConfiguration.GatewayDHCPService.Pool
		$n.VPN = $EGWConfXML.EdgeGateway.Configuration.EdgegatewayServiceConfiguration.GatewayIpsecVpnService
		$n.Routing = $EGWConfXML.EdgeGateway.Configuration.EdgeGatewayServiceConfiguration.StaticRoutingService
		$n.Syslog = $EGWConfXML.EdgeGateway.Configuration.SyslogServerSettings.TenantSyslogServerSettings.SyslogServerIp
		$n.Size = $EGWConfXML.EdgeGateway.Configuration.GatewayBackingConfig
		$n.HA = $EGWConfXML.EdgeGateway.Configuration.HaEnabled
		$n.DNSRelay = $EGWConfXML.EdgeGateway.Configuration.UseDefaultRouteForDnsRelay
		Foreach ($Interface in $n.Interfaces) {
			if ($Interface.UseForDefaultRoute -eq 'true') {$n.DefaultGateway = $Interface.SubnetParticipation.Gateway}
		$n.AdvancedNetworking= $EGWConfXML.EdgeGateway.Configuration.HaEnabled = $EGWConfXML.EdgeGateway.Configuration.AdvancedNetworkingEnabled
		$n.Org = $Vdc.Org.Name
		$n.TenantId = $Vdc.Org.Id.Split(':')[3]
		$n.OrgVDC = $Vdc.Name
		$n.OrgVDCId = $Vdc.Id.Split(':')[3]
		$n.ProviderVDC = $Vdc.ProviderVDC.Name
		$n.ProviderVDCId = $Vdc.ProviderVDC.Id.Split(':')[3]
		$n.Health = Get-NSXEdgeHealth -NSXManager $NSXManager -Username $NSXUsername -Password $NSXPassword -EdgeID ($n.EdgeBacking)
		$Output += $n
	return $Output

Automate Let’s Encrypt Certificate for NSX Edge Load Balancer

NSX LBI needed public certificate for my lab to avoid issues while testing certain libraries that did not allow untrusted connections or importing private Certificate Authority.

Fortunately, there is possibility to issue free public certificates with Let’s Encrypt certificate authority. These certificates are domain validated, which means you need to own the domain for which you issue the certificate. There are three methods how the validation is done but only one can be used in fully automated mode. Why the need for automation? The issued certificates are valid only for 90 days.

To validate the domain ownership you need to publish on publicly accessible web server under the certificate FQDN a specific generated verification string. You do not actually need to publish the service (and the NSX Edge load balancer) to the internet if you do not want to – I just set up a simple webserver with a sole purposed to complete the validation challenge.

So what is the high level process?

  1. Own a domain for which you want to have the certificate.
  2. Set up publicly accessible web server and point to it a DNS record with the certificate FQDN.
  3. Generate challenge string and place it on the web server.
  4. Validate the domain and obtain the certificates.
  5. Upload the certificates to your NSX Edge Load Balancer.
  6. In 60 days repeat from #3.

There are various ways how to automate steps 3-5. I have chosen to do this on Windows with PowerShell but the same could be accomplished on Linux as there are many Let’s Encrypt clients available to chose from.

On a Windows 2012 R2 Server I installed latest Powershell 5, IIS and ACMESharp with PowerShell gallery:

save-module -name ACMESharp
install-module -name ACMESharp

Then I wrote PowerShell script that first goes through the certificate generation and then using NSX API replaces certificate of a specific load balancer.

Note that you need to supply NSX Manager credentials, Edge ID which is running the load balancer, application profile ID which the web server uses (can be easily looked up in NSX UI) and email and domain for the Let’s Encrypt generation process.

Also be aware that Let’s Encrypt has rate limit on how many times a particular certificate can be issued within 7 day period (currently 20).

$Username = "admin"
$Password = "default"
$NSXManager = ""
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = ""
$Domain = ""

## Generate random alias
$IdentAlias = 'Ident_'+([guid]::NewGuid()).ToString()
$CertAlias = 'Cert_'+([guid]::NewGuid()).ToString()

## Remove and rename old files
If (Test-Path D:\LetsEncrypt\issuer.crt.old) {Remove-Item D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key.old) {Remove-Item D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt.old) {Remove-Item D:\LetsEncrypt\cert.crt.old}

If (Test-Path D:\LetsEncrypt\issuer.crt) {Rename-Item D:\LetsEncrypt\issuer.crt D:\LetsEncrypt\issuer.crt.old}
If (Test-Path D:\LetsEncrypt\cert.key) {Rename-Item D:\LetsEncrypt\cert.key D:\LetsEncrypt\cert.key.old}
If (Test-Path D:\LetsEncrypt\cert.crt) {Rename-Item D:\LetsEncrypt\cert.crt D:\LetsEncrypt\cert.crt.old}

## Let's Encrypt specific code from
Import-Module ACMESharp
Initialize-ACMEVault -ErrorAction SilentlyContinue
New-ACMERegistration -Contacts $Email -AcceptTos
New-ACMEIdentifier -Dns $Domain -alias $IdentAlias
Complete-ACMEChallenge $IdentAlias -ChallengeType http-01 -Handler iis -HandlerParameters @{ WebSiteRef = 'Default Web Site' }
Submit-ACMEChallenge $IdentAlias -ChallengeType http-01

$Status = "pending"
Do {
	Start-Sleep -s 5
	$Status = ((Update-ACMEIdentifier $Alias -ChallengeType http-01).Challenges | Where-Object {$_.Type -eq "http-01"}).Status
Until ($Status = "valid")

New-ACMECertificate $IdentAlias -Generate -Alias $CertAlias
Submit-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportCertificatePEM D:\LetsEncrypt\cert.crt
Get-ACMECertificate $CertAlias -ExportKeyPEM D:\LetsEncrypt\cert.key
Update-ACMECertificate $CertAlias
Get-ACMECertificate $CertAlias -ExportIssuerPEM D:\LetsEncrypt\issuer.crt

$IssuerCert = [IO.File]::ReadAllText("D:\LetsEncrypt\issuer.crt")
$PrivateKey = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.key")
$LBCertificate = [IO.File]::ReadAllText("D:\LetsEncrypt\cert.crt")

## Calculate Issuer Cert Thumbprint
$IssuerCertThumbprint = (Get-PfxCertificate -filepath D:\LetsEncrypt\issuer.crt).Thumbprint.ToLower()

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

## Get all Edge certificates
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/scope/" + $LBEdge
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content

## Find if Issuer Certificate already exists
$exists = $false
foreach ($Certificate in $sxml.certificates.certificate) {
	$Thumbprint = $Certificate.x509Certificate.sha1Hash -replace '[:]'
	if ($Thumbprint -eq $IssuerCertThumbprint) { $exists = $true }

##Upload issuer certificate if it does not exist
if (-Not $exists) {
	$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
	$Body = "
 <pemEncoding>" +$IssuerCert+ "</pemEncoding>
 <description>Issuer Certificate</description>
	$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
	$IssuerId = ([xml]$r).certificates.certificate.objectId
##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
 <pemEncoding>" + $LBCertificate + "</pemEncoding>
 <privateKey>" + $PrivateKey + "</privateKey> 
 <description>vCloud Certificate</description>
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop