vCloud Director – Storage IOPS Management

It is a little known fact that besides compute (capacity and performance), storage capacity and external network throughput rate, vCloud Director can also manage storage IOPS (input / output or read and write operations per second) performance at provisioned virtual disk granularity. This post summarizes the current capabilities.

Cloud providers usually offer different tiers of storage that is available to tenants for consumption. IOPS management helps them to differentiate these tiers and enforce the virtual disk performance based on IOPS metric. This eliminates noisy neighbor problem, but also makes both consumption and capacity management more predictive.

vCloud Director relies on vSphere to control the maximum IOPS a VM has access to on particular storage policy through a Storage I/O Control functionality which is supported on VMFS (block) and NFS datastores (no vSAN). In vSphere this is defined at virtual hard disk level, but is enforced at VM level. vSphere however does not manage available IOPS capacity of a datastore the same way it can do with compute. That’s where vCloud Director comes in.

The cloud provider first needs to create a new vSphere custom field (iopsCapacity) and use it do define for vCloud Director managed datastore their IOPS capacity. This is done via vCenter Managed Browser Object UI and is described in KB 2148300.

Definition of Custom Field iopsCapacity in vCenter MOB UI
Configuring datastore IOPS capacity in vCenter MOB UI

vCloud Director consumes vSphere datastores through storage policies. In my case I have tag based storage policy named: 2_IOPS/GB and as the name suggests the intention is to provide two provisioned IOPS per each GB of capacity. 40 GB hard disk thus should provide 80 IOPS.

Once the storage policy is synced with vCloud Director we can add it to a Provider VDC and consume it in its Org VDCs. vCloud Director will keep track of the storage policy IOPS capacity and how much has been allocated. That information is available with vCloud API when retrieving the Provider VDC storage profile representation:

Note that the pvdcStorageProfile IopsCapacity is the total IopsCapacity for all datastores as tagged in vCenter belonging to the storage policy.

The actual definition of storage policy parameters is done via PUT call at Org VDC level again with API on the Org VDC storage profile representation. The cloud provider supplies IopsSetting element that consists of the following parameters:

  • Enabled: True if this storage profile is IOPS-based placement enabled.
  • DiskIopsMax: the max IOPS that can be given to any disk (value 0 means unlimited)
  • DiskIopsDefault: the default IOPS given to any/all disks associated with this VdcStorageProfile if user doesn’t specify one
  • StorageProfileIopsLimit: the max IOPS that can be used by this VdcStorageProfile. In other words: maximum IOPS that can be assigned across all disks associated with this VdcStorageProfile (use 0 for unlimited).
  • DiskIopsPerGbMax: similar to DiskIopsMax but instead of a specific value, it’s the ratio of size (in GB) to IOPS. if set to 1, then a 1 GB disk is limited to 1 IOPS, if set to 10, then a 1 GB disk is limited to 10 IOPS, etc.

When a user deploys a VM utilizing IOPS enabled storage policy she can set specific requested IOPS for each disk though API (0 is treated as unlimited), or set nothing and vCloud Director will set default limit based on DiskIopsDefault or DiskIopsPerGbMax x DiskSizeInGb value, whichever is lower. The requested value must always be smaller than DiskIopsMax and also smaller than DiskIopsPerGbMax x DiskSizeInGb. The DiskIopsMax and DiskIopsDefault values must also be lower that StorageProfileIopsLimit.

In my case I wanted always to set IOPS limit to 2 IOPS per GB, so I configured Org VDC storage policy in the following way:

And this is provisioned VM as seen in vCloud Director UI

and in vCenter UI.

Additional observations:

  • Datastore clusters cannot be used together with IOPS storage policies. The reason is that when datastore clusters are used it is vCenter who is responsible for placing the disk to a specific datastore and as mentioned above, vCenter does not track IOPS capacity at datastore level, whereas the vCloud Director placement engine will take into account both the datastore capacity (GB) and IOPS capacity when finding the suitable datastore for a disk.
  • vSAN is not supported as it does not support SIOC. vSAN advanced storage policies allow specifying IOPS limits per object and can be used instead.
  • Disk IOPS can be assigned only to regular VMs, not to VM templates.
  • The disk IOPS will be always allocated against the Org VDC storage profile even if the VM is powered-off. This means the cloud provider can oversubscribe IOPS at the provider VDC storage profile level.
  • System administrator can override IOPS limits when deploying/editing tenant VMs in the system context.
  • Some vCloud Director versions have bug where the UI sends 0 (unlimited) IOPS for disk instead of null (undefined) which might result in provisioning error if it is not compliant with the policy limit.

20 thoughts on “vCloud Director – Storage IOPS Management

  1. Hi Tom,

    Great post! very useful to be able to limit IOPS per GB!

    I did the setup in our dev infrastructure but I mean I have found a bug… I have a policy 10_IOPS_per_GB and if I create a VM with 40 GB of disks, at the vSphere level the disk IOPS limit is set to 400 on the disk, that’s fine. But if I add a second disk to this VM, at the vSphere level the disk IOPS limit is set to Unlimited (Disk has the right Storage policy in vSphere but no limit is applied).

    Are you able to reproduce this issue ?

      1. Hi Tom
        Thanks for your feedback, I’m testing version 9.7.014029997 but our production runs always with 9.5.

  2. Hi Tomas,
    Thx a lot for the article, lot of good stuffs ! and it works.
    I have a question about the IOPS based on Gb, it seems we need to apply it for each org VDC.
    Can we do it on the providerVdcStorageProfile for once ?
    Best regards

  3. Hi, is this feature based on SIOC ?
    We have just discovered by vmware support that vCD 10 don’t support SIOC…

    1. I understand now, ‘host-based’ data services SIOC” is not supported by vCD, the support was right in my case.
      So the feature in this page is a the legacy tag based storage policie, we need to tag datastore and do some call API in order to set the right iops per gb, am i right ?

  4. Hello Tom

    Great post, thx !

    I have some questions:

    I was wondering it this was still the case for 10.1 as from version 10.1 host based services as encryption is supported?

    Would a tenant be able to change the IO’s for a certain disk without increasing the disk size?
    In the policy you would put DiskIopsMax 1000 and DiskIopsPerGbMax at 5.

    When he deploys a disk of 100GB, he would have 500 IOPS.
    Would he then be able to overrule the IOPS limit on the disk to a value below the max, let’s says 750?

    1. No change with 10.1. This IOPS management method does not rely on host based IOPS services storage policies.
      Yes, tenant can change disk IOPS config as long as it is lower than the defined DiskIopsMax.

  5. Hello Tom,

    We notice in our environment big differences between virtual machines
    Some do 1/3 of the IOP’s but twice the throughput regarding MB/s.

    Storage policies in vSan default to a block size of 32KB.
    A vm running having a storage profile at 1000 iops and running 32 kb would get 1000 iops
    A vm running having a storage profile at 1000 iops ans running 64 kb would get 500 iops
    Both would ending up the same throughput which is the end-result you want, and as policy this is also very clear to the customer.

    When using the storage policies for 3-Tier storage in vcd, can you also limit the vmdk on throughput?

  6. It may be helpful to someone – you can set the IOPSCapacity value using a web request in PowerShell, for example:

    $vc_server = “vc_server_ip”

    $moref = (Get-CIDatastore -Name “datastore_name” -Server “vcd_server”).ExtensionData.VimObjectRef.moref
    $moref = $moref -replace “-“,”%2d”

    $mob_URI = “https://$vc_server/mob/?moid=$moref&method=setCustomValue”

    function Set-VCIOPSCapacity(){
    Param(
    [Parameter(Mandatory=$True)] [string] $mob_URI
    )

    $vc_username = “vc_user”
    $vc_password = “vc_password”
    $secpasswd = ConvertTo-SecureString $vc_password -AsPlainText -Force
    $credential = New-Object System.Management.Automation.PSCredential($vc_username, $secpasswd)

    # Ingore SSL Warnings
    add-type -TypeDefinition @”
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
    public bool CheckValidationResult(
    ServicePoint srvPoint, X509Certificate certificate,
    WebRequest request, int certificateProblem) {
    return true;
    }
    }
    “@
    [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
    $webReq = Invoke-WebRequest -Uri $mob_URI -Credential $credential -Verbose -SessionVariable vmware
    $null = $webReq -match ‘name=”vmware-session-nonce” type=”hidden” value=”?([^\s^”]+)”‘
    $sessionnonce = $matches[1]
    $body = @”
    vmware-session-nonce=$sessionnonce&key=iopsCapacity&value=”1000000″
    “@
    Invoke-WebRequest -Uri $mob_URI -Credential $cred -Verbose -Method Post -Body $body -WebSession $vmware
    }

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.