vCloud Availability: Replication of Powered-off VM

Just a short post about a feature I recently learned.

In vSphere Replication when you are configuring replication of powered-off VM you will get the following message:

The virtual machine is not powered on. Replication will start when the virtual machine is powered on.

The replication is actually configured and its placeholder VM is created in the recovery location (cloud) but the VM will stay in Not Active state.

Why is this? Immediate start of replication locks VM disks which means such VM would not be able to power-on until the initial sync is finished. But what if you want to replicate powered-off VMs for example templates that are never meant to run?

You can in fact force start the replication by right clicking the VM and selecting Sync Now, which asks confirmation question if we really want to do so as the VM will not be able the be powered on until the operation completes.

Is there a use case for this? As I mentioned this could be used for catalog sync as replication is much faster and efficient that OVF export / import.

Advertisements

vCloud Architecture Toolkit for Service Provider Update

The vCloud Architecture Toolkit for Service Provider website has been updated with new set of documents. All documents were re-branded with the new VMware Cloud Provider Program logos that replace the old vCloud Air Network brand.

My Architecting a VMware vCloud Director Solution for VMware Cloud Providers whitepaper has been refreshed to include vCloud Director 8.10 and 8.20 additions that were missing in the previous version. The current version of the document is 2.8 with August 2017 release date.

Here is summary of the new or updated topics:

  • Cell sizing
  • vCloud DB performance tips
  • New vCenter Chargeback Manager network metrics
  • vRealize Business for Cloud
  • vRealize Log Insight
  • vRealize Operations Manager
  • NSX Networking updates
  • Storage support
  • vCloud RBAC
  • Org VDC vSphere Resource Settings
  • VCDNI deprecation
  • New Org VDC Edge GW features
  • Distributed Firewall
  • VM Auto import
  • vCloud API for NSX
  • vCloud Director orchestrated upgrade

The document can be downloaded in PDF format or viewed online.

vCloud Director 8.20: VM Auto-import

While the biggest new feature of vCloud Director 8.20 is the access for tenants to NSX advanced services on Edge Gateways and distributed firewall, service providers will love automatic import of vCenter virtual machines.

Import of vCenter VMs under vCloud Director management was around for long time, but the VM had to be in powered off state. In vCloud Director 8.10, the possibility of running VM import was introduced (vCloud API only feature). However, vCloud Director 8.20 simplifies this even more:

The system administrator can just drag any vCenter VM into Org VDC resource pool and vCloud Director will automatically discover such VM and import it. The VM can be even in running state. This enables migration use cases, or allows service providers to easily offer self service access to their fully managed vSphere only environments by simply connecting them to vCloud Director.

In the picture below you can see Org VDC Resource Pool (ACME_PAYG) in vCenter hierarchy with two regular vCenter VMs that were dragged there. VM_C1 was running, while VM_P2 was powered-off.

resource-pool

The next picture shows automatically created vApps for these two imported VMs with the prefix Discovered.

vapps

While these vApps resemble regular vApps they are not real vCloud Director vApps until they are adopted. The adoption happens when the VM inside the vApp is somehow reconfigured.

By default VM discovery is enabled for every Organization in vCloud Director. It can be disabled in General Settings (UI or API)

general-settings

or with CMT command on VCD cell:

cell-management-tool manage-config -n managed-vapp.discovery.activated -v false

This behavior can be overridden at Org VDC level with API element <VmDiscoveryEnabled>.

Here are the differences between Discovered and Adopted vApp:

Discovered vApp

  • Looks like a regular vApp
  • Can have only one VM per discovered vApp
  • vApp contains API element <autoNature>true</autoNature>
  • When imported VM is deleted in VC or VCD, its vApp object will get automatically purged
  • Is owned by system
  • Is not subject to Org lease settings

Adopted vApp

  • Regular vCloud Director vApp
  • Can contain multiple VMs, vApp networks, etc.
  • Discovered vApp will get adopted when it is reconfigured (other than changing its name or description)

Other Considerations

  • Discovery process runs in the background every 3 mins
  • Failed VM import is retried after 60 mins. This can be changed with CMT command (example for 25 seconds):
    cell-management-tool manage-config -n managed-vapp.discovery.retry-delay-sec -v 25
  • The following VMs cannot be imported: Fault Tolerant VMs, VMs with creation/upload process in VC, templates, VCD shell VMs
  • VM must be connected to Org VDC network.
  • VM can be running or powered off.
  • VM does not need to use Org VDC storage policy. If it resides on unknown storage policy, it is automatically relocated to the default Org VDC storage policy during the adoption however the VM must be in powered off state.
  • VM with IDE controller must be in powered off state.
  • VM CPU/RAM resources are changed based on Org VDC allocation type.
  • VM resources are not subject to Org VDC allocation restrictions, but are charged against it.
  • VM name in VC remains intact until it is adopted and renamed
  • New vSphere 6.5 guest operating systems are not recognized and are imported as Other (32-bit) OS.

Update 4/24/2017

  • By default there is minimal VM age configured to 1 hour. It means VMs that were freshly reconfigured will be skipped for the import. This is to ‘settle’ the VMs first. The interval can however be changed with CMT command (age set to 60 seconds):
    cell-management-tool manage-config -n VM_DISCOVERY_MIN_AGE_SEC -v 60
  • Related to the minimal age, it is important that all cells and vCloud database are using the same time configuration. Not only the time must be correct but also the time zone must be configured identically.

Monitoring vSphere Replication RPO Compliance

Just a quick post to show how you can monitor Recovery Point Objective (RPO) compliance of a virtual machines protected with vSphere Replication.

Option 1: vCenter Server Alarm

When vSphere Replication Appliance is registered to vCenter Server multiple new vSphere Replication Event Types become available and can be used for creation of custom alarms.

List of all these event types can be queried with the following one-line PowerCLI command:

(get-view eventManager).get_Description()| select -expand Eventinfo |where FullFormat -like “*Hms*”

The following example will show how to set alarm for event “RPO violated”

Key:ExtendedEvent
Description: RPO violated
Category: error
FullFormat: com.vmware.vcHms.rpoViolatedEvent|Virtual machine vSphere Replication RPO is violated by [data.currentRpoViolation] minute(s)

  1. In vCenter Server go to Manager, Alarm Definitions and add new alarm
  2. Set alarm name, monitor VMs and specific events.
    new-alarm
  3. Enter the trigger (com.vmware.vcHms.rpoViolatedEvent)
    alarm-trigger
  4. Add Alarm actions (email, SNMP trap, run command etc.) as necessary.

Triggered alarm:

triggered-alarm

Note that this alarm applies only to VMs replicated from the particular vCenter Server. So it will not be triggered on VMs replicated to this vCenter Server.

Option 2: vCloud API

This options applies only for VM replications to or from a cloud provider who uses vCloud Availability add-on. The vCloud Director tenant APIs are extended with replication APIs. The state of each replication can be retrieved with:

GET /api/vr/replications/<replication-id>

and

GET /api/vr/failbackreplications/<replication-id>

Where list of all replications and their replication-ids is retrieved at org level with these two API calls:

GET /api/org/<org-id>/replications

and

GET /api/org/<org-id>/failbackreplications

An example of VM1 replication state (RPO 15 mins, not active with 16 min RPO violation):

replication-api

The following tables describes all the elements of the API response:

replication-details

vCloud Availability PowerCLI Monitoring Cmdlets

vCloud Availability PowerCLI Monitoring Cmdlets


vCloud Availability for vCloud Director
is a new product for vCAN service providers released in July 2016 that enables disaster recovery to and from the cloud. It allows any vSphere customer who installes vSphere Replication to protect her on-prem workloads into the multitenant service provider cloud environment.

vCloud Availability

vCloud Availability adds additional vCloud APIs thanks to its vCloud Extensibility Framework. I have created a few simple PowerCLI cmdlets that demonstrate how easy is to use the new APIs. The cmdlets allow service provider to monitor state of vCloud Availability components as well as state of tenant replications.

Get-VRCells

Discovers all vCloud Director cells (including Cloud Proxies) and reports their status.

Get-VRCells

Get-VRCSNodes

Discovers all vSphere Replication Cloud Service nodes, their version and amount of seconds since last successful heartbeat.

Get-VRCSNodes

Get-VRMServers

Discovers all vSphere Replication Manager Servers and reports their status and paired vCenter Server

Get-VRMServers

Get-VRServers

Discovers all vSphere Replication Servers, their status and statistics about utilization.

Get-VRServers

Get-VRReplications

Collects all replications in particular organization, their compliance status and other information (RPO, storage used, number of snapshots, etc.). This cmdlet can be used also with tenant credentials.

Get-VRReplications

Function Get-VRCells {
<#
.SYNOPSIS
Discovers all vCloud Director cells
.DESCRIPTION
Discovers all vCloud Director cells including Cloud Proxies and report their status and version and if they ran VC proxy .EXAMPLE PS C:\> Get-VRCells
.NOTES
Author: Tomas Fojta
#>

if (-not $global:DefaultCIServers) {Connect-CIServer}

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "query?type=cell"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$VRCells = $sxml.QueryResultRecords.CellRecord


$VRCells
}

Function Get-VRCSNodes {
<#
.SYNOPSIS
Discovers all vSphere Replication Cloud Service nodes
.DESCRIPTION
Discovers all vSphere Replication Cloud Service nodes, their version and last heartbeat
.EXAMPLE
PS C:\> Get-VRCSNodes
.NOTES
Author: Tomas Fojta
#>
if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRCSNodes = @()
$CurrentTime = (New-TimeSpan -Start (Get-Date -Date "01/01/1970") -End (Get-Date).ToUniversalTime()).TotalMilliseconds

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/vr/nodes"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($VRCS in $sxml.VrcsNodes.Node) {
$n = @{} | select VRCS,MaxSupportedAPiVersion,LastHeartbeatTimestamp,LastHeartbeatAgoInSeconds
$n.VRCS = $VRCS.Id
$n.MaxSupportedAPiVersion = $VRCS.MaxSupportedAPiVersion
$n.LastHeartbeatTimestamp = $VRCS.LastHeartbeatTimestamp
$n.LastHeartbeatAgoInSeconds = [math]::Truncate(($CurrentTime - [convert]::ToInt64($VRCS.LastHeartbeatTimestamp))/1000)

$VRCSNodes += $n
}

$VRCSNodes
}

Function Get-VRMServers {
<#
.SYNOPSIS
Discovers all vSphere Replication Manager Servers and reports their status
.DESCRIPTION
Discovers all vSphere Replication Manager Servers, reports their status and paired vCenter Server
.EXAMPLE
PS C:\> Get-VRMServers
.NOTES
Author: Tomas Fojta
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$false,Position=1)]
[String]$Org
)
if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRMServers = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/extension/vimServerReferences"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($VimServer in $sxml.VMWVimServerReferences.VimServerReference) {
$n = @{} | select VRMS,IsConnected,VIMName,VIMhref
$n.VIMname = $VimServer.name
$n.VIMhref = $VimServer.href

$r = Invoke-WebRequest -URI ($VimServer.href + "/vrmServer") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$n.VRMS = $sxml.AdminVrmServer.Uuid
$n.IsConnected = $sxml.AdminVrmServer.IsConnected

$VRMServers += $n
}

$VRMServers
}

Function Get-VRServers {
<#
.SYNOPSIS
Collects all vSphere Replication Servers
.DESCRIPTION
Collects all vSphere Replication Servers and basic statistics
.EXAMPLE
PS C:\> Get-VRServers
.NOTES
Author: Tomas Fojta
#>

if (-not $global:DefaultCIServers) {Connect-CIServer}

$VRServers = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "admin/extension/vr/vrServers"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($vRS in $sxml.References.Reference) {
$n = @{} | select Name,href,IsConnected,ReplicationsCount,DisksCount
$n.href = $vRS.href

$r = Invoke-WebRequest -URI $vRS.href -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Name = $sxml.AdminVrServer.ReplicationTrafficAddress
$n.IsConnected = $sxml.AdminVrServer.IsConnected

$r = Invoke-WebRequest -URI ($vRS.href+"/statistics") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content
$n.ReplicationsCount = $sxml.VrServerStatistics.ReplicationsCount
$n.DisksCount = $sxml.VrServerStatistics.DisksCount

$VRServers += $n
}

$VRServers

}

Function Get-VRReplications {
<#
.SYNOPSIS
Collects all replications in particular organization
.DESCRIPTION
Collects all replications in particular organization, their compliance status and other information
.EXAMPLE
PS C:\> Connect-CIServer -Org ACME
PS C:\> Get-VRReplications
.EXAMPLE
PS C:\> Get-VRReplications -Org ACME
.NOTES
Author: Tomas Fojta
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$false,Position=1)]
[String]$Org
)
if (-not $global:DefaultCIServers) {Connect-CIServer}
If ($Org -eq "") {$Org = $global:DefaultCIServers[0].Org}

$VRReplications = @()

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + ((get-org $org).id).Replace("urn:vcloud:org:","org/") + "/replications"
$head = @{"x-vcloud-authorization"=$global:DefaultCIServers[0].SessionSecret} + @{"Accept"="application/*+xml;version=20.0;vr-version=3.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

foreach ($Replication in $sxml.References.Reference) {
$n = @{} | select Name,href,Rpo,ReplicationState,CurrentRpoViolation,TestRecoveryState,RecoveryState,VappId,VrServerInfo,ReplicaSpaceRequirements, Instance
$n.href = $Replication.href

$r = Invoke-WebRequest -URI $Replication.href -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Rpo = $sxml.ReplicationGroup.Rpo
$n.ReplicationState = $sxml.ReplicationGroup.ReplicationState
$n.CurrentRpoViolation = $sxml.ReplicationGroup.CurrentRpoViolation
$n.TestRecoveryState = $sxml.ReplicationGroup.TestRecoveryState
$n.RecoveryState = $sxml.ReplicationGroup.RecoveryState
$n.VappId = $sxml.ReplicationGroup.PlaceholderVappId.Replace("urn:vcloud:vapp:","")
$n.VrServerInfo = $sxml.ReplicationGroup.VrServerInfo.Uuid

$r = Invoke-WebRequest -URI ($Replication.href + "/instances") -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.ReplicaSpaceRequirements = $sxml.ReplicationGroupInstanceList.ReplicaSpaceRequirements
$n.Instance = $sxml.ReplicationGroupInstanceList.Instance

$Uri = $global:DefaultCIServers[0].ServiceUri.AbsoluteUri + "vApp/vapp-" + $n.VappId
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$n.Name = $sxml.VApp.Children.Vm.name

$VRReplications += $n
}

$VRReplications

}