VCD Cell Management Tool without Administrator Credentials

I just learned from engineering neat trick related to how cell management tool can be invoked without specifying administrator credentials.

The issue is that currently you cannot use LDAP account to trigger cell management tool commands which are mostly used for quiescing and shutting down cells for maintenance. Using vCloud Director local administrator account is discouraged as it poses a security issue. However what is possible is to trigger the cell management tool as root (or with sudo) and supply via hidden flag -i the process ID of the java process.

Here is an example:

PID

First I query the java PID with ps aux command. Then I use the standard cell-management-tool command without specifying the user with the -i flag at the end.

So you can force the administrator to log in to the cell guest OS via a LDAP account and then run the command with sudo.

Thank you Zachary Shepherd for the tip.

Custom vCenter Server Event and Alarm

Related to my previous post about monitoring Edge Gateways my customer asked me if he could leverage vCenter Server alarms as they are integrated with their monitoring and alerting infrastructure.

So basically is there a way to create vCenter alarm via scripted action (for example with PowerCLI)?

The answer is yes and it is not that difficult. There are two types of vCenter alarms: based on condition/state or an event. And it is possible to create custom user event via Loguserevent method of entity manager via vSphere API.

This is example of PowerCLI code the creates user event “Edge Gateway event” at the root datacenter folder.

$DCFolderEntity = Get-Folder -Name datacenters
$eventMgr = Get-View EventManager
$eventMgr.LogUserEvent($entity.ExtensionData.MoRef,"Edge Gateway event")

User logged event

Now it is easy to create custom alarm based on the user logged event. Create the alarm at the vCenter Server root level, as alarm type pick monitor vCenter Server and as triggers manually enter (type):

Event; vim.event.GeneralUserEvent 

That’s it.

Alarm

Renaming Edge Gateway

Recently one of my customers changed naming convention of vCloud Director Edge Gateways. He renamed them in vCloud Director however the names of virtual machines backing the Edge Gateways stayed the same. Also the names in vShield Manager or NSX Manager did not change. He quickly found out that there is no way to change the name in the GUI of vShield/NSX Manager. This is unfortunate especially for troubleshooting purposes.

Just today I saw the same question internally on our mailing list and the answer is that you can rename the Edge via API. Just GET the particular edge configuration and PUT it back with the changed name. As I recently wrote Powershell scripts that leveraged NSX API I quckly modified it and created two functions – RenameEdgeV5 and RenameEdgeV6.

They both rename the Edge, however only the first one works with vShield Manager. When using NSX Manager you have to use the one that matches the Edge version. The legacy Edges deployed by vCloud Director are version 5. The new NSX Edges are version 6. The API calls to retrieve the Edge info and change it are very similar one uses api 3.0 the other api 4.0.

Here are the scripts:

function RenameEdgeV5 {
<#
.SYNOPSIS Renames vShield Edge
.DESCRIPTION Renames vShield or NSX legacy v5 Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your vShield or NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
.EXAMPLE
PS> RenameEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId -Name newname
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId,
[Parameter(Mandatory=$true)]
[String]$Name
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/3.0/edges"+"/"+$EdgeId
$s = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $s.Content
$sxml.edge.name = $Name
$r = Invoke-WebRequest -Uri $HealthRequest -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

return $r.StatusCode
function RenameEdgeV6 {
<#
.SYNOPSIS Renames NSX Edge
.DESCRIPTION Renames NSX Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
.EXAMPLE
PS> RenameEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId -Name newname
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId,
[Parameter(Mandatory=$true)]
[String]$Name
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/4.0/edges"+"/"+$EdgeId
$s = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $s.Content
$sxml.edge.name = $Name
$r = Invoke-WebRequest -Uri $HealthRequest -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

return $r.StatusCode

} # End of process

} # End of function

Usage example:Usage

vSphere Client recent task activity:
Recent Tasks

 

And NSX Manager Edge Gateway view.RenamedEdges

My New Role at VMware

After almost 4 years working for professional services at VMware I am transferring to new Architect role in Global Cloud Practice – vCloud Air Network. I am taking up the challenge to work in a global team of cloud architects with half of them VCDX certified which will support the largest vCloud Air Network Service Provider partners.

During my PSO days as Consulting Architect I was consulting, designing and implementing private and public clouds for global, large and also smaller customers all over Europe with most of them in German-speaking countries. Whenever I encountered an interesting problem I blogged about it on this blog and recently the most often topics blogged about were about vCloud Director and NSX.

So I feel it is a perfect fit for me that In the new role I will specialize on public cloud only for handful of large SPs which means you can expect more articles about vCloud Director and NSX :-).

Stay tuned…

How to Monitor Health of NSX Edge Gateways

NSX Edge Service Gateways are virtual machines deployed by NSX Manager that provide network services (routing, bridging, load balancing, VPNs, DNS relay, DHCP, …). This makes them quite a critical component in the infrastructure and thus there might be a need to keep a close eye on their availability.

While NSX Manager reports the status of the Edges in the GUI and logs it might take some time to register a change in the health. If you want up-to-date status of an Edge health you need to query the NSX Manager with NSX API. The NSX Manager then retrieves the current status of the Edge. The mechanism of the communication between NSX Manager and Edge appliance(s) depends on the Edge version and the vSphere cluster status:

VIX Communication

This is the legacy mode of communication. NSX Manager utilizes VIX API to query vCenter Server and the ESXi Host which runs the Edge appliance who then via VM Tools talks to the actual VM. This mode of communication is used for the legacy Edges version 5.5.x (deployed via  the compatibility vShield v2 API) and as failback mode when for some reason Message Bus Communication mode is not possible.

 

Message Bus Communication

This is direct (and faster) communication between NSX Manager and the ESXi host (vsfwd process) running the Edge appliance. During the Edge deployment the cluster where the Edge is deployed to must be prepared for NSX and without any issues.

Message Bus

source: NSXvSphereDesignGuidev2.1.pdf

 

To query the Edge health is an expensive operation – it takes time and creates load on NSX Manager. If there is large number of Edges (for example in service provider scenario) this should be considered.

To test the viability of checking at least once in a given time the status of all Edges health I have created simple Powershell function Get-NSXEdgeHealth:

function Get-NSXEdgeHealth {
<#
.SYNOPSIS Gathers Health Status of a NSX Edge
.DESCRIPTION Will query NSX Manager for the health of a particular NSX Edge
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.PARAMETER EdgeId
ID of the Edge to gather health data for.
.EXAMPLE
PS> Get-NSXEdge -NSXManager nsxmgr.fqdn -Username admin -Password password -EdgeId EdgeId
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password,
[Parameter(Mandatory=$true)]
[String]$EdgeId
)
Process {
### Ignore TLS/SSL errors
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
$HealthRequest = "https://$NSXManager/api/3.0/edges"+"/"+$EdgeId+"/status"

$h = @{} | select Health, Detail

$r = Invoke-WebRequest -Uri $HealthRequest -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$rxml = $r.Content
$h.Health = $rxml.edgeStatus.edgeStatus
$Details = @()
foreach ($Feature in $rxml.edgeStatus.featureStatuses.featureStatus)
{
$n = @{} | select Service, Status
$n.Service = $Feature.service
$n.Status = $Feature.status
$Details += $n
}
$h.Detail = $Details

return ,$h

} # End of process

PowerShell 3.0 or higher and (at least audit) credentials (and connectivity) to NSX Manager are needed.

Usage example:

Example

 

As can be seen the function needs the Edge ID parameter and then returns the overall Edge health and also detailed status of all its network services.

Following health states are defined:

  • green – good. This is the only state that guarantees that the Edge is functional.
  • red – no backing appliance is in service state
  • grey – unknown status (for example undeployed Edge)
  • yellow – intermittent health check failures (if more than 5 consecutive health checks fail the status goes to red)

Following function Get-NSXEdges will collect all Edges in the environment:

function Get-NSXEdges {
<#
.SYNOPSIS Gathers NSX Edges from NSX Manager
.DESCRIPTION Will inventory all of your NSX Edges from NSX Manager
.NOTES Author: Tomas Fojta
.PARAMETER NSXManager
The FQDN or IP of your NSX Manager
.PARAMETER Username
The username to connect with. Defaults to admin if nothing is provided.
.PARAMETER Password
The password to connect with
.EXAMPLE
PS> Get-NSXEdges -NSXManager nsxmgr.fqdn -Username admin -Password password
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,Position=0)]
[String]$NSXManager,
[Parameter(Mandatory=$false,Position=1)]
[String]$Username = "admin",
[Parameter(Mandatory=$true)]
[String]$Password
)

Process {
### Ignore TLS/SSL errors
add-type @"cd
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
### Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}
### Connect to NSX Manager via API
$Request = "https://$NSXManager/api/3.0/edges"
$r = Invoke-WebRequest -Uri ($Request+"?startIndex=0&pageSize=1") -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$TotalCount = ([xml]$r).pagedEdgeList.edgePage.pagingInfo.totalCount

$r = Invoke-WebRequest -Uri ($Request+"?startIndex=0&pageSize="+$TotalCount) -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$rxml = $r.Content

### Return the NSX Edges

$Edges = @()

foreach ($EdgeSummary in $rxml.pagedEdgeList.edgePage.edgeSummary)
{
$n = @{} | select Name,Id
$n.Name = $edgeSummary.Name
$n.Id = $edgeSummary.objectId
$Edges += $n
}
return ,$Edges

} # End of process

} # End of function

And here is a sample script leveraging both functions above that continuously displays health status of all the Edges in the environment and also displays the time needed to go through all of them.

$NSXManager = "nsx01.fojta.com"
$Username = "admin"
$Password = "default"

$AllEdges = Get-NSXEdges -NSXManager $NSXManager -Username $Username -Password $Password

DO
{
$StartTime = get-date
foreach ($Edge in $AllEdges)
{
$Health = Get-NSXEdgeHealth -NSXManager $NSXManager -Username $Username -Password $Password -EdgeId $Edge.Id
Write-Host $Edge.Name $Health.Health
}
$Duration = (get-date) - $StartTime
Write-Host
Write-Host "Duration:" $Duration.Minutes "Minutes" $Duration.Seconds "Seconds"
Write-Host
} While ($true)

In my lab it took at least cca 2 seconds to get status of an Edge (depending on the mode of communication and its actual health). It is obvious that most time NSX Manager spends on communication with the ESXi host – so the task should and can be parallelized. While running 5 sessions at the same time the retrieval of health status of one Edge went up to 3-4 seconds (for a green Edge) while the load on NSX Manager went up 14 % (I run NSX Manager only with 2 vCPUs in my lab).

Monitoring

While the article mentions only NSX the scripts should work also with vShield / vCloud Networking and Security Manager.