vCloud Director – Storage IOPS Management

It is a little known fact that besides compute (capacity and performance), storage capacity and external network throughput rate, vCloud Director can also manage storage IOPS (input / output or read and write operations per second) performance at provisioned virtual disk granularity. This post summarizes the current capabilities.

Cloud providers usually offer different tiers of storage that is available to tenants for consumption. IOPS management helps them to differentiate these tiers and enforce the virtual disk performance based on IOPS metric. This eliminates noisy neighbor problem, but also makes both consumption and capacity management more predictive.

vCloud Director relies on vSphere to control the maximum IOPS a VM has access to on particular storage policy through a Storage I/O Control functionality which is supported on VMFS (block) and NFS datastores (no vSAN). In vSphere this is defined at virtual hard disk level, but is enforced at VM level. vSphere however does not manage available IOPS capacity of a datastore the same way it can do with compute. That’s where vCloud Director comes in.

The cloud provider first needs to create a new vSphere custom field (iopsCapacity) and use it do define for vCloud Director managed datastore their IOPS capacity. This is done via vCenter Managed Browser Object UI and is described in KB 2148300.

Definition of Custom Field iopsCapacity in vCenter MOB UI
Configuring datastore IOPS capacity in vCenter MOB UI

vCloud Director consumes vSphere datastores through storage policies. In my case I have tag based storage policy named: 2_IOPS/GB and as the name suggests the intention is to provide two provisioned IOPS per each GB of capacity. 40 GB hard disk thus should provide 80 IOPS.

Once the storage policy is synced with vCloud Director we can add it to a Provider VDC and consume it in its Org VDCs. vCloud Director will keep track of the storage policy IOPS capacity and how much has been allocated. That information is available with vCloud API when retrieving the Provider VDC storage profile representation:

Note that the pvdcStorageProfile IopsCapacity is the total IopsCapacity for all datastores as tagged in vCenter belonging to the storage policy.

The actual definition of storage policy parameters is done via PUT call at Org VDC level again with API on the Org VDC storage profile representation. The cloud provider supplies IopsSetting element that consists of the following parameters:

  • Enabled: True if this storage profile is IOPS-based placement enabled.
  • DiskIopsMax: the max IOPS that can be given to any disk (value 0 means unlimited)
  • DiskIopsDefault: the default IOPS given to any/all disks associated with this VdcStorageProfile if user doesn’t specify one
  • StorageProfileIopsLimit: the max IOPS that can be used by this VdcStorageProfile. In other words: maximum IOPS that can be assigned across all disks associated with this VdcStorageProfile
  • DiskIopsPerGbMax: similar to DiskIopsMax but instead of a specific value, it’s the ratio of size (in GB) to IOPS. if set to 1, then a 1 GB disk is limited to 1 IOPS, if set to 10, then a 1 GB disk is limited to 10 IOPS, etc.

When a user deploys a VM utilizing IOPS enabled storage policy she can set specific requested IOPS for each disk though API (0 is treated as unlimited), or set nothing and vCloud Director will set default limit based on DiskIopsDefault or DiskIopsPerGbMax x DiskSizeInGb value, whichever is lower. The requested value must always be smaller than DiskIopsMax and also smaller than DiskIopsPerGbMax x DiskSizeInGb. The DiskIopsMax and DiskIopsDefault values must also be lower that StorageProfileIopsLimit.

In my case I wanted always to set IOPS limit to 2 IOPS per GB, so I configured Org VDC storage policy in the following way:

And this is provisioned VM as seen in vCloud Director UI

and in vCenter UI.

Additional observations:

  • Datastore clusters cannot be used together with IOPS storage policies. The reason is that when datastore clusters are used it is vCenter who is responsible for placing the disk to a specific datastore and as mentioned above, vCenter does not track IOPS capacity at datastore level, whereas the vCloud Director placement engine will take into account both the datastore capacity (GB) and IOPS capacity when finding the suitable datastore for a disk.
  • vSAN is not supported as it does not support SIOC. vSAN advanced storage policies allow specifying IOPS limits per object and can be used instead.
  • Disk IOPS can be assigned only to regular VMs, not to VM templates.
  • The disk IOPS will be always allocated against the Org VDC storage profile even if the VM is powered-off. This means the cloud provide can oversubscribe IOPS at the provider VDC storage profile level.
  • System administrator can override IOPS limits when deploying/editing tenant VMs in the system context.
  • Some vCloud Director versions have bug where the UI sends 0 (unlimited) IOPS for disk instead of null (undefined) which might result in provisioning error if it is not compliant with the policy limit.

Resource Consumption of Org VDC Allocation Types

The reference table below summarizes how different vCloud Director Org VDC allocation types consume vSphere resources. In other words: how a choice of allocation model for a particular Org VDC and its parameters (allocation, guarantees, quota, vCPU speed) translate to resource pool and VM resource settings (CPU/RAM) – reservations and limits.

click for large version

Notes:

  • valid for vCloud Director 9.5 and older (down to 5.5)
  • RP … resource pool
  • Elastic … Org VDC can be divided across multiple RPs across clusters
  • Although there are currently only three Org VDC allocation types the Allocation Pool can be elastic or non-elastic based on vCloud Director instance wide setting in General Settings

vCloud Director – vCenter Server Relationship

vCloud Director as a cloud management platform needs resources to provision the target workloads to. These resources are provided by vCenter Server (compute, storage, networks) and NSX Manager (networks and networking services).

In the past vCloud Director required tight grip on those resources, so the recommendation and best practice was to dedicate them to the particular vCloud Director instance. System admins were discouraged to run additional workloads not managed by vCloud Director on them. However that has changed recently, therefore the need for this blog article.

vCenter Server Extension

vCloud Director used to register itself as a vCenter Server extension. That allowed to ‘protect’ VCD managed VMs with a special icon and a warning pop up during vCenter edits of such VMs.

vCloud Director specific VM icon

Today, vCloud Director is quite resilient against changes on a particular VM done directly in vCenter Server, so there is no more need for those warnings. vCloud Director 9.5 thus no longer register itself as vCenter Server extension, so you will no longer see these icons and pop up warnings.

As a side note you will also see a change in the VM naming. The long UUID is no longer added to the VM name and is replaced by shorter 4 digit random characters.

Host Preparation

In the past during creation of a Provider VDC, the system admin was asked for ESXi host credentials. This was needed to upload cloud agent vib that was used for certain features (thumbnails, VCDNI network encapsulation). All these features were either replaced by different mechanism or deprecated, so there is no need to upload any vcloud vibs to ESXi hosts anymore.

Additionally, custom attribute system.service… used to be set for each vCloud Director managed host and vCloud Director managed VM. This provided a way to control where vCenter DRS could vMotion VMs through this host to VM compatibility option. Disabling a host would remove the custom property. vCloud Director VM could not be vMotioned to unprepared host as vCenter would scream the host is incompatible with the VM.

 

Host Custom Attributes
VM Custom Attribute

vMotion to Unprepared Host Error

In vCloud Director 9.5 this mechanism was completely eliminated. When host is put into maintenance mode, it is considered unavailable for vCloud Director, therefore there is no need anymore to disable it first in vCloud Director. You will  no longer see any host preparation dialog and the hosts section is simplified to bare minimum.

So What About the Relationship?

As you can see the vCloud Director – vCenter Server relationship is very loose. In fact it is no longer monogamous, meaning vCenter Server can be married (associated) with multiple vCloud Director instances at the same time.

Why would you do that?

I can think of three use cases, but obviously our smart service providers will come up with more.

Use case 1: Test & Dev

Do you need to test new vCloud Director release? Or provide test instance of vCloud Director for your internal developers? No need to spin up whole vSphere + NSX environment with storage, etc. Just deploy one VM with vCloud Director bits (you can even use the appliance if you have external DB and NFS ready) and point it to the existing vSphere/NSX endpoints.

Use case 2: Whitelabeling / Reselling

To enable three tier mode, where SP provides infrastructure, reseller will get its own vCloud Director instance (branded) to resell to their end customers. SP needs to setup one big vSphere/NSX infrastructure and have an automated way to deploy VCD instances on top of it. The reseller gets its own instance with system admin equivalent rights and manages its own tenants.

Use case 3: Uber Org Admin Role

Some end-customers request bigger than Org Admin role. They want to create their own organizations and Org VDCs to better align with their business groups. SP can dedicate whole VCD instance to such customer. Without the need of provisioning dedicated vSphere/NSX as well.

Any Caveats, Recommendations?

  • Segment vSphere environment with Clusters and Resource Pools for each VCD instance.
  • Use different VCD instance names and IDs
  • Use separate accounts both for vCenter Server and NSX for each VCD instance. Give each account permission only to resources it should see (use vCenter No-Access privilege on clusters/RP/folders it should not see)
  • Dedicate storage resources to each VCD
  • Use separate NSX transport zone for each VCD instance
  • Monitor the load of multiple VCD listeners on the single VC. Scale out VCs if needed. VMware does not test this kind of setup at scale.

Fun Fact

You can in fact vMotion running VM from one vCloud Director environment to another one. To do so, you will move it in vCenter from the source Org VDC RP to the destination Org VDC RP. You must also move it out of the VM Folder (remember the No Access privilege?) to be visible to the target VCD. Obviously it needs to be connected to the right target networks.

Finally, you will need to remove the original vCloud UUID (with PowerCLI or similar) and let it be auto-discovered by the target VCD. There is no auto-removal from the original VCD, so you will need to use the process described here.

 

 

 

vCloud Director vApp Runtime Lease Expiration Action

In vCloud Director it is possible to configure vApp leases. The maximums are set by system admin at Organization level (in Policies), which can be lowered by Org Admin (at org level) and set by vApp owner at the vApp level. A vApp has runtime lease (for how long it will be in running state) and storage lease (for how long it will consume storage once it is not running).

vApp leases are very useful in test & dev or lab environments to make sure abandoned, unused VMs are not running and taking resources.

When vApp lease is coming to an end, its owner gets a reminder via email (how many days before expiration can be configured in User Preferences) and can optionally reset vApp lease to avoid its stopping or deletion.

By default expired running vApp is put into suspended state which means its memory content is saved to datastores. This ensures fully consistent state upon consequent power on of the vApp. This however might not be always needed especially in dev/lab situations – the memory content could take lots of storage space and for example saving 16 GB RAM VM to datastore could also create IO performance impact. As of vCloud Director 8.20 the Organization Administrator can instead change the default runtime expiry action to power off. The setting is done at Org level and must be done via API by setting the element <PowerOffOnRuntimeLeaseExpiration> of OrgLeaseSettingsType to true. The API version must be at least 25.0.

PUT /api/admin/org/eea1f10c-3fee-43d7-bd8e-be63453d6e34/settings/vAppLeaseSettings

Accept:application/*+xml;version=29.0
x-vcloud-authorization:{{x-vcloud-authorization}}
Content-Type:application/vnd.vmware.admin.vAppLeaseSettings+xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<VAppLeaseSettings xmlns="http://www.vmware.com/vcloud/v1.5">
    <DeleteOnStorageLeaseExpiration>true</DeleteOnStorageLeaseExpiration>
    <DeploymentLeaseSeconds>1209600</DeploymentLeaseSeconds>
    <StorageLeaseSeconds>15552000</StorageLeaseSeconds>
    <PowerOffOnRuntimeLeaseExpiration>true</PowerOffOnRuntimeLeaseExpiration>
</VAppLeaseSettings>

Note:

When the vApp expiry action is set to power off, the actual VM stop action power off (hard) vs shutdown (gracefull) procedure depends on the vApp’s config for each VM (tab Starting and Stopping VMs).

Also note that subsequent edit of Org policies in UI will reset the Org PowerOffOnRuntimeLeaseExpiration setting back to default (false).

Org VDC Edge Gateway CPU/RAM Reservations

vCloud Director 8.20 allows deployment of Org VDC Edge Gateways in 4 different form factors from Compact to X-Large where each provides different level of performance and consumes different amount of resources.

As these Edge Gateways are deployed by NSX Manager which allows setting custom reservations for CPU and RAM via an API call PUT https://<NSXManager>/api/4.0/edgePublish/tuningConfiguration, it is also possible in vCloud Director to set custom reservations.

Why would you change the default reservations? Reservations at VM (Edge) level reserve the resources for itself which means no other VM can utilize them in case they are unused. They basically guarantee certain level of service that the VM (Edge) from performance perspective will always deliver. In service provider environments oversubscription provides ROI benefits and if the service provider can guarantee enough resources at cluster scale, than the VM level reservations can be set lower if at all.

This can be accomplished by tuning the networking.gatewayMemoryReservationMultiplier and networking.gatewayCpuReservationMultiplier settings via cell-management-tool from vCloud Director cell. By default the CPU multiplier is set to 64 MHz per vCPU and the Memory multiplier to 0.5.

By default Edge Gateways will be deployed with the following reservation settings:

Org VDC Edge GW Default Resource Reservations

The following command will change memory multiplier to 10%:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n networking.gatewayMemoryReservationMultiplier -v 0.1

Note: The new reservation settings are applicable only for newly deployed Org VDC Edge Gateways. Redeploying existing edges will not change their reservation settings. You must either use NSX API to do so, or modify Org VDC Edge Gateway form factor (e.g. change Large to Compact and then back to Large) which is not so elegant as it will basically redeploy the Edge twice.

Also note that NSX 6.2 and NSX 6.3 have different sizing of Quad Large Edge. vCloud Director 8.20 is by default set for the NSX 6.3 size which is 2 GB RAM (as opposed to NSX 6.2 value of 1 GB RAM). It is possible to change the default for the reservation calculation by editing networking.full4GatewayMemoryMb setting to value ‘1024’