vCloud Director 8.20: VM Auto-import

While the biggest new feature of vCloud Director 8.20 is the access for tenants to NSX advanced services on Edge Gateways and distributed firewall, service providers will love automatic import of vCenter virtual machines.

Import of vCenter VMs under vCloud Director management was around for long time, but the VM had to be in powered off state. In vCloud Director 8.10, the possibility of running VM import was introduced (vCloud API only feature). However, vCloud Director 8.20 simplifies this even more:

The system administrator can just drag any vCenter VM into Org VDC resource pool and vCloud Director will automatically discover such VM and import it. The VM can be even in running state. This enables migration use cases, or allows service providers to easily offer self service access to their fully managed vSphere only environments by simply connecting them to vCloud Director.

In the picture below you can see Org VDC Resource Pool (ACME_PAYG) in vCenter hierarchy with two regular vCenter VMs that were dragged there. VM_C1 was running, while VM_P2 was powered-off.

resource-pool

The next picture shows automatically created vApps for these two imported VMs with the prefix Discovered.

vapps

While these vApps resemble regular vApps they are not real vCloud Director vApps until they are adopted. The adoption happens when the VM inside the vApp is somehow reconfigured.

By default VM discovery is enabled for every Organization in vCloud Director. It can be disabled in General Settings (UI or API)

general-settings

or with CMT command on VCD cell:

cell-management-tool manage-config -n managed-vapp.discovery.activated -v false

This behavior can be overridden at Org VDC level with API element <VmDiscoveryEnabled>.

Here are the differences between Discovered and Adopted vApp:

Discovered vApp

  • Looks like a regular vApp
  • Can have only one VM per discovered vApp
  • vApp contains API element <autoNature>true</autoNature>
  • When imported VM is deleted in VC or VCD, its vApp object will get automatically purged
  • Is owned by system
  • Is not subject to Org lease settings

Adopted vApp

  • Regular vCloud Director vApp
  • Can contain multiple VMs, vApp networks, etc.
  • Discovered vApp will get adopted when it is reconfigured (other than changing its name or description)

Other Considerations

  • Discovery process runs in the background every 3 mins
  • Failed VM import is retried after 60 mins. This can be changed with CMT command (example for 25 seconds):
    cell-management-tool manage-config -n managed-vapp.discovery.retry-delay-sec -v 25
  • The following VMs cannot be imported: Fault Tolerant VMs, VMs with creation/upload process in VC, templates, VCD shell VMs
  • VM must be connected to Org VDC network.
  • VM can be running or powered off.
  • VM does not need to use Org VDC storage policy. If it resides on unknown storage policy, it is automatically relocated to the default Org VDC storage policy during the adoption however the VM must be in powered off state.
  • VM with IDE controller must be in powered off state.
  • VM CPU/RAM resources are changed based on Org VDC allocation type.
  • VM resources are not subject to Org VDC allocation restrictions, but are charged against it.
  • VM name in VC remains intact until it is adopted and renamed
  • New vSphere 6.5 guest operating systems are not recognized and are imported as Other (32-bit) OS.

Monitoring vSphere Replication RPO Compliance

Just a quick post to show how you can monitor Recovery Point Objective (RPO) compliance of a virtual machines protected with vSphere Replication.

Option 1: vCenter Server Alarm

When vSphere Replication Appliance is registered to vCenter Server multiple new vSphere Replication Event Types become available and can be used for creation of custom alarms.

List of all these event types can be queried with the following one-line PowerCLI command:

(get-view eventManager).get_Description()| select -expand Eventinfo |where FullFormat -like “*Hms*”

The following example will show how to set alarm for event “RPO violated”

Key:ExtendedEvent
Description: RPO violated
Category: error
FullFormat: com.vmware.vcHms.rpoViolatedEvent|Virtual machine vSphere Replication RPO is violated by [data.currentRpoViolation] minute(s)

  1. In vCenter Server go to Manager, Alarm Definitions and add new alarm
  2. Set alarm name, monitor VMs and specific events.
    new-alarm
  3. Enter the trigger (com.vmware.vcHms.rpoViolatedEvent)
    alarm-trigger
  4. Add Alarm actions (email, SNMP trap, run command etc.) as necessary.

Triggered alarm:

triggered-alarm

Note that this alarm applies only to VMs replicated from the particular vCenter Server. So it will not be triggered on VMs replicated to this vCenter Server.

Option 2: vCloud API

This options applies only for VM replications to or from a cloud provider who uses vCloud Availability add-on. The vCloud Director tenant APIs are extended with replication APIs. The state of each replication can be retrieved with:

GET /api/vr/replications/<replication-id>

and

GET /api/vr/failbackreplications/<replication-id>

Where list of all replications and their replication-ids is retrieved at org level with these two API calls:

GET /api/org/<org-id>/replications

and

GET /api/org/<org-id>/failbackreplications

An example of VM1 replication state (RPO 15 mins, not active with 16 min RPO violation):

replication-api

The following tables describes all the elements of the API response:

replication-details

vCloud Availability PowerCLI Monitoring Cmdlets

vCloud Availability PowerCLI Monitoring Cmdlets


vCloud Availability for vCloud Director
is a new product for vCAN service providers released in July 2016 that enables disaster recovery to and from the cloud. It allows any vSphere customer who installes vSphere Replication to protect her on-prem workloads into the multitenant service provider cloud environment.

vCloud Availability

vCloud Availability adds additional vCloud APIs thanks to its vCloud Extensibility Framework. I have created a few simple PowerCLI cmdlets that demonstrate how easy is to use the new APIs. The cmdlets allow service provider to monitor state of vCloud Availability components as well as state of tenant replications.

Get-VRCells

Discovers all vCloud Director cells (including Cloud Proxies) and reports their status.

Get-VRCells

Get-VRCSNodes

Discovers all vSphere Replication Cloud Service nodes, their version and amount of seconds since last successful heartbeat.

Get-VRCSNodes

Get-VRMServers

Discovers all vSphere Replication Manager Servers and reports their status and paired vCenter Server

Get-VRMServers

Get-VRServers

Discovers all vSphere Replication Servers, their status and statistics about utilization.

Get-VRServers

Get-VRReplications

Collects all replications in particular organization, their compliance status and other information (RPO, storage used, number of snapshots, etc.). This cmdlet can be used also with tenant credentials.

Get-VRReplications

Function Get-VRCells {
<#
.SYNOPSIS
Discovers all vCloud Director cells
.DESCRIPTION
Discovers all vCloud Director cells including Cloud Proxies and report their status and version and if they ran VC proxy .EXAMPLE PS C:\> Get-VRCells
.NOTES
Author: Tomas Fojta
#>

if (-not $global:DefaultCIServers) {Connect-CIServer}

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "query?type=cell"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$VRCells = $sxml.QueryResultRecords.CellRecord


$VRCells
}

Function Get-VRCSNodes {
<#
.SYNOPSIS
Discovers all vSphere Replication Cloud Service nodes
.DESCRIPTION
Discovers all vSphere Replication Cloud Service nodes, their version and last heartbeat
.EXAMPLE
PS C:\> Get-VRCSNodes
.NOTES
Author: Tomas Fojta
#>
if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRCSNodes = @()
$CurrentTime = (New-TimeSpan -Start (Get-Date -Date "01/01/1970") -End (Get-Date).ToUniversalTime()).TotalMilliseconds

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/vr/nodes"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($VRCS in $sxml.VrcsNodes.Node) {
$n = @{} | select VRCS,MaxSupportedAPiVersion,LastHeartbeatTimestamp,LastHeartbeatAgoInSeconds
$n.VRCS = $VRCS.Id
$n.MaxSupportedAPiVersion = $VRCS.MaxSupportedAPiVersion
$n.LastHeartbeatTimestamp = $VRCS.LastHeartbeatTimestamp
$n.LastHeartbeatAgoInSeconds = [math]::Truncate(($CurrentTime - [convert]::ToInt64($VRCS.LastHeartbeatTimestamp))/1000)

$VRCSNodes += $n
}

$VRCSNodes
}

Function Get-VRMServers {
<#
.SYNOPSIS
Discovers all vSphere Replication Manager Servers and reports their status
.DESCRIPTION
Discovers all vSphere Replication Manager Servers, reports their status and paired vCenter Server
.EXAMPLE
PS C:\> Get-VRMServers
.NOTES
Author: Tomas Fojta
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$false,Position=1)]
[String]$Org
)
if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRMServers = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/extension/vimServerReferences"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($VimServer in $sxml.VMWVimServerReferences.VimServerReference) {
$n = @{} | select VRMS,IsConnected,VIMName,VIMhref
$n.VIMname = $VimServer.name
$n.VIMhref = $VimServer.href

$r = Invoke-WebRequest -URI ($VimServer.href + "/vrmServer") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$n.VRMS = $sxml.AdminVrmServer.Uuid
$n.IsConnected = $sxml.AdminVrmServer.IsConnected

$VRMServers += $n
}

$VRMServers
}

Function Get-VRServers {
<#
.SYNOPSIS
Collects all vSphere Replication Servers
.DESCRIPTION
Collects all vSphere Replication Servers and basic statistics
.EXAMPLE
PS C:\> Get-VRServers
.NOTES
Author: Tomas Fojta
#>

if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRServers = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/extension/vr/vrServers"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($vRS in $sxml.References.Reference) {
$n = @{} | select Name,href,IsConnected,ReplicationsCount,DisksCount
$n.href = $vRS.href

$r = Invoke-WebRequest -URI $vRS.href -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Name = $sxml.AdminVrServer.ReplicationTrafficAddress
$n.IsConnected = $sxml.AdminVrServer.IsConnected

$r = Invoke-WebRequest -URI ($vRS.href+"/statistics") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$n.ReplicationsCount = $sxml.VrServerStatistics.ReplicationsCount
$n.DisksCount = $sxml.VrServerStatistics.DisksCount

$VRServers += $n
}

$VRServers

}

Function Get-VRReplications {
<#
.SYNOPSIS
Collects all replications in particular organization
.DESCRIPTION
Collects all replications in particular organization, their compliance status and other information
.EXAMPLE
PS C:\> Connect-CIServer -Org ACME
PS C:\> Get-VRReplications
.EXAMPLE
PS C:\> Get-VRReplications -Org ACME
.NOTES
Author: Tomas Fojta
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$false,Position=1)]
[String]$Org
)
if (-not $global:DefaultCIServers) {Connect-CIServer}
If ($Org -eq "") {$Org = $global:DefaultCIServers[0].Org}

$VRReplications = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + ((get-org $org).id).Replace("urn:vcloud:org:","org/") + "/replications"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($Replication in $sxml.References.Reference) {
$n = @{} | select Name,href,Rpo,ReplicationState,CurrentRpoViolation,TestRecoveryState,RecoveryState,VappId,VrServerInfo,ReplicaSpaceRequirements, Instance
$n.href = $Replication.href

$r = Invoke-WebRequest -URI $Replication.href -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Rpo = $sxml.ReplicationGroup.Rpo
$n.ReplicationState = $sxml.ReplicationGroup.ReplicationState
$n.CurrentRpoViolation = $sxml.ReplicationGroup.CurrentRpoViolation
$n.TestRecoveryState = $sxml.ReplicationGroup.TestRecoveryState
$n.RecoveryState = $sxml.ReplicationGroup.RecoveryState
$n.VappId = $sxml.ReplicationGroup.PlaceholderVappId.Replace("urn:vcloud:vapp:","")
$n.VrServerInfo = $sxml.ReplicationGroup.VrServerInfo.Uuid

$r = Invoke-WebRequest -URI ($Replication.href + "/instances") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.ReplicaSpaceRequirements = $sxml.ReplicationGroupInstanceList.ReplicaSpaceRequirements
$n.Instance = $sxml.ReplicationGroupInstanceList.Instance

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "vApp/vapp-" + $n.VappId
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Name = $sxml.VApp.Children.Vm.name

$VRReplications += $n
}

$VRReplications

}

 

Query ESXi Hosts Serial Numbers

I was asked by our IT department to provide serial numbers of our lab servers. Fortunately this can be done remotely with esxcfg-info CLI command and can be automated with William Lam (@lamw) PowerCLI function Get-Esxcfginfo. I just had to find the right entry in the xml file returned by the function.

Here is the script I used:

<#
.SYNOPSIS Remoting collecting esxcfg-info from an ESXi host using vCenter Server
.NOTES Author: William Lam
.NOTES Site: www.virtuallyghetto.com
.NOTES Reference: http://www.virtuallyghetto.com/2016/06/using-the-vsphere-api-to-remotely-collect-esxi-esxcfg-info.html
.PARAMETER Vmhost
 ESXi host
.EXAMPLE
 PS> Get-VMHost -Name "esxi-1" | Get-Esxcfginfo
#>

Function Get-Esxcfginfo {
 param(
 [Parameter(
 Position=0,
 Mandatory=$true,
 ValueFromPipeline=$true,
 ValueFromPipelineByPropertyName=$true)
 ]
 [VMware.VimAutomation.ViCore.Impl.V1.Inventory.InventoryItemImpl[]]$VMHost
 )

 $sessionManager = Get-View ($global:DefaultVIServer.ExtensionData.Content.sessionManager)

 # URL to the ESXi esxcfg-info info
 $url = "https://" + $vmhost.Name + "/cgi-bin/esxcfg-info.cgi?xml"

 $spec = New-Object VMware.Vim.SessionManagerHttpServiceRequestSpec
 $spec.Method = "httpGet"
 $spec.Url = $url
 $ticket = $sessionManager.AcquireGenericServiceTicket($spec)

 # Append the cookie generated from VC
 $websession = New-Object Microsoft.PowerShell.Commands.WebRequestSession
 $cookie = New-Object System.Net.Cookie
 $cookie.Name = "vmware_cgi_ticket"
 $cookie.Value = $ticket.id
 $cookie.Domain = $vmhost.name
 $websession.Cookies.Add($cookie)

 # Retrieve file
 $result = Invoke-WebRequest -Uri $url -WebSession $websession -ContentType "application/xml"
 
 # cast output as an XML object
 return [ xml]$result.content
}

Connect-VIServer -Server xxx.gcp.local -User administrator@vsphere.local -password VMware1! | Out-Null

$hosts = Get-VMHost

foreach ($ESXhost in $hosts)
{
$xmlResult = $ESXhost | Get-Esxcfginfo
Write-Host $ESXhost.name ($xmlResult.host.'hardware-info'.value[3].'#text')
}

Disconnect-VIServer * -Confirm:$false

host-serials

NSX L2 Bridging Options

I had recently multiple discussions about NSX and its Layer 2 bridging capabilities with various service providers. Let me summarize some important points and considerations when you would use which.

Why?

Let’s start with simple question – why would you need layer 2 bridging? Here are some use cases:

  • The end-user wants to burst their application to the cloud but wants to keep certain components on-site and because its legacy application it cannot be re-IP’d or requires single subnet communication.
  • The service provider is building new cloud offering next to legacy business (collocation, managed services) and wants to enable existing customers to migrate or extend their workloads to the cloud seamlessly (no IP address changes)

How?

NSX offers three ways how to bridge layer two networks.

Layer 2 VPN

This is proprietary VPN solution which enables to create encrypted tunnel across IP networks between Edge Service Gateways that stitches one or more L2 networks. These Edge Gateways can be deployed in different management domains and there is also option of deploying standalone Edge which does not require NSX license. This is great for the cloud bursting use case. I have blogged about L2VPN already in the past here.

While this option is very flexible it is also quite CPU intensive for both the L2 VPN Client and Server Edge VMs. This option provides up to 2Gb throughput.

NSX Native L2 Bridging

L2 bridge is created in the ESXi VMkernel hypervisor by deploying a Logical router control VM. The control VM is used only for the bridge configuration and its pinning to a particular ESXi host. As the bridging happens in the VMkernel it is possible to achieve impressive line rate (10 Gb) throughput.

It is possible to bridge only VXLAN based logical switch with VLAN based port group. The same physical uplink must be utilized so this means that the VLAN port group must be on the same vSphere Distributed Switch (vDS) that is prepared with the VXLAN VTEP and where the VXLAN logical switch portgroups are created.

L2Bridge

VLAN and VXLAN portgroups are on the same vDS
VLAN and VXLAN portgroups are on the same vDS

The above fact prohibits a scenario where you would have Edge Cluster with multiple pairs of uplinks connected to separate vDS switches. One for VLAN based traffic and the other for VXLAN traffic. You cannot create NSX native L2 bridge instance between two vDS switches.

This important especially for the collocation use case mentioned at the beginning. In order to use the L2 bridge the customer VLAN must be connected to the Edge Cluster Top of the Rack pair of switches.

If this is not possible, as a workaround the service provider can use the L2 VPN option – it is even possible to run both L2 VPN Server and Client Edges on the same host connected through a transit VXLAN network where one Edge is connected to trunk with VLAN networks from one vDS and the other to trunk with VXLAN networks on another vDS. Unfortunately this has performance impact (even if NULL-MD5 encryption is used) and should be used only for temporary migration use cases.

L2VPN interfaces

L2VPN

Hardware VTEP

The last bridging option discussed is a new feature of NSX  6.2. It is possible to extend the VXLAN logical switch all the way to a compatible hardware device (switch from a VMware partner) that acts as Layer 2 gateway and bridges the logical switch with a VLAN network. The device performing the function of hardware VTEP is managed from NSX via OVSDB protocol while the control plane is still managed by NSX Controllers. More details are in the following white paper.

As this option requires new dedicated and NSX compatible switching hardware it is more useful for the permanent use cases.