VMware Cloud Director – Storage IOPS Management – Part II

This is a follow up to the article I posted about a year ago that describes new IOPS management functionality in VMware Cloud Director (VCD) 10.2.

Storage IOPS  is next to compute, networking and storage capacity a limited resource service providers want to manage in order to fairly share underlying physical resources in a multitenant environment.

As was described in the original article VCD supported storage IOPS management  however the feature was quite hidden and available only via API. The recent release of VMware Cloud Director not only fully exposes the functionality in the UI but also adds some new functionality. Let’s dive into it.

There are two main mechanisms now how you can manage IOPS.

vCenter Server managed IOPS

This mechanism relies on setting IOPS limits at storage policy level directly in vCenter Server. That is possible with host based and with vSAN based storage policies. This mechanism is quite simple – when a VM disk is provisioned to such IOPS limited storage policy it will inherit the IOPS limit –  a constant number per policy. You will not be able to set proportional IOPS based on disk capacity.

vSAN Storage Policy with IOPS Limit

Host Based non vSAN Storage Policy with IOPS Limit

I would recommend using such mechanism only if you want to avoid noisy neighbors. The concept is not new, VCD could use such vSAN policies for some time and host based policies were already supported in VCD 10.1. The only difference is that now in 10.2 the tenant will see the limit reservation set at VM disk level but will not be able to change it.

Non-editable Disk IOPS

VCD Managed IOPS

This is much more sophisticated mechanism where you can really manage IOPS as pool of available capacity that you slice and allocate to tenant Org VDCs. This is the mechanism that was until now only available via API.

You will start by tagging your datastores with their IOPS capacity – that has not changed and still must be done from within VC via custom properties.

At Provider VDC level you can then create IOPS managed storage policies and define their service level in terms of disk IOPS defaults, maximums or IOPS allocation based on disk size (0 means unlimited).

This storage policy configuration can be inherited or overridden at Org VDC level. This is big improvement compared to the old approach where you had to create such storage policies always at Org VDC level.

Another new thing is that you can disable IOPS placement mechanism for such storage policy. This is useful in case you want to use Datastore Clusters. VCD will no longer try to place each virtual disk based on a particular datastore available IOPS. The placement decision is instead done by vCenter Server – you should therefore enable Storage DRS with I/O balancing automation. There is no need in such case to tag individual datastores in VC with their IOPS capacity.

Some of the old caveats still apply:

• Disk IOPS can be assigned only to regular VMs or named (independent) disks, not to VM templates.
• The disk IOPS will be always allocated against the Org VDC storage profile even if the VM is powered-off. This means the cloud provider can oversubscribe IOPS at the provider VDC storage profile level.
• System administrator can override IOPS limits when deploying/editing tenant VMs in the system context.

vSAN File Services with vCloud Director

vSphere 7 is now generally available and with it came also new vSAN update that introduces vSAN File Service. Cormac Hogan has good overview of the feature on his blog so definitely head there first to understand what it is.

I want to dive into the possibility of using vSAN File Service NFS in vCloud Director environments.

Let me start with current (April 2020) interop – vSphere 7 is not supported yet with vCloud Director. Which means vCenter Server 7 cannot be used as a target for IaaS services. But that is not an issue for the use case I want to discuss today.

vCloud Director deployment needs NFS storage for its Transfer Share directory. vCloud Director architecture consists of multiple cells (VM nodes) that scale out horizontally based on the size of the managed environment. The cells need shared database and shared Transfer Share directory to function properly. The Transfer Share must be NFS mount and is used mostly for OVF import/export operations related to vApp template and catalog management however the appliance deployment mode of vCloud Director also uses transfer share for storing appliance related info, ssh keys, responses.properties file for deployment of additional cells, and embedded database backups.

vCloud Director cell VMs are usually deployed in the management cluster and that can be separate vSphere 7 environment with vSAN. Can we (or should we) use vSAN NFS for vCloud Director Transfer Share?

Current practice is either to use external hardware storage NFS (NetApp) or to deploy Linux VM with large disk that acts as NFS server. The first approach is not always possible especially if you use vSAN only and have no external storage available. Then you have to go with the Linux VM approach. But not anymore.

 

vSAN File Service NFS has the following advantages:

• no external dependency on hardware storage or Linux VM
• easy to deploy and manage from UI or programmatically
• capacity management with quotas and thresholds
• high availability
• integrated lifecycle

The whole end-to-end deployment is indeed very simple, let me demonstrate the whole process:

1. Start with vSAN FS configuration in vSphere Cluster > Configure > vSAN > Services > File Service
2. Directly download vSAN File service agent (the lightweight container image OVA)
3. Configure vSAN domain and networking
   
   
4. Provide pool of IP addresses for the containers (I used 4 as I have 4 host management cluster).
   
5. After while you will see the agent containers deployed on each cluster node.
   
6. Now we can proceed with NFS share configuration. In the vSphere Cluster > Configure > vSAN > File Service Shares > ADD. We can define name, vSAN storage policy and quotas.
   
7. Enter IP addresses of your vCloud Director cells to grant them access to this share. Set permission Read/Write and make sure root squash is disabled.
   
8. Once the share is created, select the check box and copy its URL. Chose the NFSv4.1.
   In my case it looks like 192.168.110.181:/vsanfs/VCDTransferShare
9. Now you use the string in your vCloud Director cell deployment. I am using the vCloud Director appliance.
   
10. Once the cell is started we can see how the transfer share is mounted:
    
    Notice that while the mount IP address in /etc/fstab is the provided one 192.168.110.171, the actual one used is 192.168.110.172. This provides load balancing across all service node when more exports are created and when NFSv4.1 mount address is used.

It is imported to understand that although we have 4 vSAN FS agents deployed, the TransferShare will be provided via single container – in my case it is the one with IP 192.168.110.172. To find out on which host this particular container is running go to Cluster > Monitor > vSAN > Skyline Health > File Service > File Service Health.

So what happens if the host esx-03a.corp.local is unavailable? The share will fail over to another host. This took in my tests around 60-90 seconds. During that time the NFS share will not be accessible but the mount should persist and once the failover finishes it will become accessible again.

Notice that 192.168.110.172 is now served from esx-04.corp.local.

Also note that maintenance mode of the host will not vMotion the agent. It will just shut it down (and after while undeploy) and rely on the above mechanism to fail over the share to another agent. You should never treat the vSAN FS agents as regular VMs.

I am excited to see vSAN File Services as another piece of VMware technology that removes 3rd party dependencies from running vCloud Director (as was the case with NSX load balancing and PostgreSQL embedded database).

vCloud Director Object Storage Extension Reference Design

Just a quick announcement that a vCloud Director Object Storage Extension Reference Design that I wrote is now available at this link.

It deep dives into the use cases, architecture, includes recommended deployment options and description of the new feature of 1.0.1 related to multisite deployments. There are also results of performance tests of the overhead the extension adds over the direct use of the native storage platform.

vCloud Director – Storage IOPS Management

Update 2020/10/22 Make sure to read part II for updates.

It is a little known fact that besides compute (capacity and performance), storage capacity and external network throughput rate, vCloud Director can also manage storage IOPS (input / output or read and write operations per second) performance at provisioned virtual disk granularity. This post summarizes the current capabilities.

Cloud providers usually offer different tiers of storage that is available to tenants for consumption. IOPS management helps them to differentiate these tiers and enforce the virtual disk performance based on IOPS metric. This eliminates noisy neighbor problem, but also makes both consumption and capacity management more predictive.

vCloud Director relies on vSphere to control the maximum IOPS a VM has access to on particular storage policy through a Storage I/O Control functionality which is supported on VMFS (block) and NFS datastores (no vSAN). In vSphere this is defined at virtual hard disk level, but is enforced at VM level. vSphere however does not manage available IOPS capacity of a datastore the same way it can do with compute. That’s where vCloud Director comes in.

The cloud provider first needs to create a new vSphere custom field (iopsCapacity) and use it do define for vCloud Director managed datastore their IOPS capacity. This is done via vCenter Managed Browser Object UI and is described in KB 2148300.

Definition of Custom Field iopsCapacity in vCenter MOB UI

Configuring datastore IOPS capacity in vCenter MOB UI

vCloud Director consumes vSphere datastores through storage policies. In my case I have tag based storage policy named: 2_IOPS/GB and as the name suggests the intention is to provide two provisioned IOPS per each GB of capacity. 40 GB hard disk thus should provide 80 IOPS.

Once the storage policy is synced with vCloud Director we can add it to a Provider VDC and consume it in its Org VDCs. vCloud Director will keep track of the storage policy IOPS capacity and how much has been allocated. That information is available with vCloud API when retrieving the Provider VDC storage profile representation:

Note that the pvdcStorageProfile IopsCapacity is the total IopsCapacity for all datastores as tagged in vCenter belonging to the storage policy.

The actual definition of storage policy parameters is done via PUT call at Org VDC level again with API on the Org VDC storage profile representation. The cloud provider supplies IopsSetting element that consists of the following parameters:

• Enabled: True if this storage profile is IOPS-based placement enabled.
• DiskIopsMax: the max IOPS that can be given to any disk (value 0 means unlimited)
• DiskIopsDefault: the default IOPS given to any/all disks associated with this VdcStorageProfile if user doesn’t specify one
• StorageProfileIopsLimit: the max IOPS that can be used by this VdcStorageProfile. In other words: maximum IOPS that can be assigned across all disks associated with this VdcStorageProfile (use 0 for unlimited).
• DiskIopsPerGbMax: similar to DiskIopsMax but instead of a specific value, it’s the ratio of size (in GB) to IOPS. if set to 1, then a 1 GB disk is limited to 1 IOPS, if set to 10, then a 1 GB disk is limited to 10 IOPS, etc.

When a user deploys a VM utilizing IOPS enabled storage policy she can set specific requested IOPS for each disk though API (0 is treated as unlimited), or set nothing and vCloud Director will set default limit based on DiskIopsDefault or DiskIopsPerGbMax x DiskSizeInGb value, whichever is lower. The requested value must always be smaller than DiskIopsMax and also smaller than DiskIopsPerGbMax x DiskSizeInGb. The DiskIopsMax and DiskIopsDefault values must also be lower that StorageProfileIopsLimit.

In my case I wanted always to set IOPS limit to 2 IOPS per GB, so I configured Org VDC storage policy in the following way:

And this is provisioned VM as seen in vCloud Director UI

and in vCenter UI.

Additional observations:

• Datastore clusters cannot be used together with IOPS storage policies. The reason is that when datastore clusters are used it is vCenter who is responsible for placing the disk to a specific datastore and as mentioned above, vCenter does not track IOPS capacity at datastore level, whereas the vCloud Director placement engine will take into account both the datastore capacity (GB) and IOPS capacity when finding the suitable datastore for a disk.
• vSAN is not supported as it does not support SIOC. vSAN advanced storage policies allow specifying IOPS limits per object and can be used instead.
  
• Disk IOPS can be assigned only to regular VMs, not to VM templates.
• The disk IOPS will be always allocated against the Org VDC storage profile even if the VM is powered-off. This means the cloud provider can oversubscribe IOPS at the provider VDC storage profile level.
• System administrator can override IOPS limits when deploying/editing tenant VMs in the system context.
• Some vCloud Director versions have bug where the UI sends 0 (unlimited) IOPS for disk instead of null (undefined) which might result in provisioning error if it is not compliant with the policy limit.

vCloud Director Object Storage Extension – Deep Look

VMware released last week another product that extends vCloud Director and enables Cloud Service Providers to offer additional services on top of vCloud Director out-of-the-box IaaS. Where vCloud Availability adds Disaster Recovery and migration services to vCloud Director, Container Service Extension adds the ability to deploy Kubernetes clusters, vRealize Operations Tenant App brings advanced workload monitoring, the newly released vCloud Director Object Storage Extension offers easy access for the tenants to a scalable, cheap, durable and network accessible storage for their applications.

As the name suggests it is an extension, that lives side by side to vCloud Director and that requires 3rd party object storage provider. In the 1.0 release the only supported storage provider is Cloudian Hyperstore, however other storage providers (cloud or on-prem) are coming in future releases. The extension provides multitenant S3 compatible API endpoint as well as user interface plugin for vCloud Director.

Use Cases

The object storage service is fully in the service provider competence who decides its parameters (SLAs, scalability) and upsells it to existing or new vCloud Director tenants.

The tenants can provision storage buckets and directly upload/download objects into them via the UI, or use S3 APIs or S3 compatible solutions to do so. Objects can be also accessible via S3 path-style URL for easy sharing.

Additionally tenants can provision application credentials and use them in their (stateless) workloads to persist application configuration or logs and have access to unstructured data (web servers).

Tight integration with vCloud Director also offers usage of object storage as archival or distribution resource for vCloud Director vApps and Catalogs. Tenant can capture existing vApps to a dedicated object storage bucket and later restore it to its Org VDCs.

Alternatively whole vCloud Director Organization Catalog of vApp templates and ISO images can be captured to the bucket or created from scratch by uploading individual ISO and OVA objects and used by same or another Organization even in a different vCloud Director instance via the catalog subscribe mechanism.

S3 API Compatibility

The solution supports S3 API with AWS Signature V4, which means existing applications can easily leverage the Object Storage service without the need for rewrites. The below screenshots show usage of S3 Browser freeware Windows client to manage the files.

Objects can be tagged and assigned with metadata, buckets can be tagged as well. Server side encryption can be configured by the Org Admin at tenant level or via API at object level. SSE-S3 (server managed key) and SSE-C (client supplied keys) methods are supported. Access Control List (ACL) permissions can be set at bucket or at object. Buckets can be shared within the tenant (to subset or all users) or made public.

Security credentials (pair of access and secret keys) are of two types. User credentials (can manage all users buckets and objects) and application credentials (can only manage subset of buckets). Object Storage Extension automatically creates user credential for each tenant user, however additional user or application credentials can be created. Credentials can be disabled and/or deleted.

The full set of supported S3 APIs is documented via the swagger UI on the extension endpoint (/docs) or here.

Provider Management

While the object storage tenant consumption APIs are standardized (S3 AWS APIs), each storage platform uses different admin APIs. Object Storage Extension currently does not expose provider APIs. The tenant administration (service entitlement) is done from the vCloud Director provider UI.

Other administration (quotas, usage metering, platform monitoring, etc.) are done directly through the Cloudian Management Console where the provider admin is redirected from the vCloud Director UI or optionally through Cloudian HyperStore Admin APIs.. This will change in later releases when more storage providers are supported.

Roles

Object Storage Service uses three different user personas. Provider administrator, tenant administrator and tenant user. Provider administrator manages tenant access to service and the storage platform. Tenant administrator has access to all buckets and objects of a particular tenant and can monitor consumption at organization, user or bucket level. Tenant user can only access her own buckets and objects or the ones shared with the user.

The user personas map to users based on their vCloud Director rights. The mapping in general corresponds to System Administrator / Organization Administrator / other non Organization Administrator global roles, unless these were changed in vCloud Director.

Provider Administrator (system context):

• General: Administrator View
• Provider VDC: View
• Organization: View
• UI Plugins: View

Tenant Administrator:

• General: Administrator View
• Organization VDC: View
• UI Plugins: View
• excludes: Provider VDC: View

Tenant User:

• UI Plugins: View
• excludes: Administrator: View

Architecture

The Object Storage Extension has 1:1 relationship with vCloud Director instance and 1:1 relationship with the storage provider (Cloudian HyperStore). Each vCloud Director Organization that is enabled to consume the service will have unique counterpart at the storage platform (Cloudian HyperStore business groups). Same is valid for users. As it is vCloud Director who provides authentication to the service, it is fully multitenant.

The diagram (taken from the official documentation) below shows all the components needed for the Object Service Extension including the traffic flows. vCloud Director 9.1 and newer is supported. Next to the vCloud Director cells you will need to deploy one or more (for HA and scalability) RHEL/CentOS/Oracle Linux VM nodes (dark green in the picture) that will run the Object Storage Extension service that is provided is RPM package. These VMs are essentially stateless and persist all their data in PostgreSQL DB. This could be vCloud Director external PostgreSQL DB (if possible) or a dedicated database just for the Object Storage Extension.

The service needs its own public IP address as it runs (by default) on port 443. S3 API clients or the vCloud Director UI plugin will access this endpoint. vCloud API extensibility is not used, but vCloud Director HTML 5 UI extensibility is.

The extension VM nodes need to have access to vCloud API endpoint for user authentication and for the vApp/Catalog import/export functionality. Additionally they will need fast access to the underlying object storage platform (in our case Cloudian HyperStore). Cloudian HyperStore is fully distributed with a minimum supported deployment of three (fully equivalent) storage nodes and scales essentially indefinitely. Each storage node also provides UI/API functionality. Fast L4 load balancing should be used to forward the extension calls to all storage nodes. Multiple APIs (S3, IAM and Admin) each running on separate TCP port need to be accessed as well as Cloudian Management Console for the Provider UI plugin redirection (this is the only service that needs to be set up with sticky sessions).

As can be seen the Object Storage Extension is in the datapath of the object transfers that are persisted on the storage nodes. The overhead is less than 10% when compared to accessing Cloudian directly (with TLS sessions) however the extension nodes must be sized properly (it is a CPU intensive workload) so they do not become a bottleneck. Both scale-out and scale-up options are possible.

The Cloudian HyperStore storage nodes can be deployed in three different configurations. For small environments or testing it can be deployed as virtual appliance running on vSphere (CentOS + HyperStore binary) leveraging shared (more expensive) or local disk storage (HyperStore replicates objects across storage nodes so it does not need highly available shared storage). Another options are to deploy Cloudian Hyperstore on dedicated bare metal hardware or to purchase hardware appliances directly from Cloudian. It is up to service provider to decide which form factor to use to tailor the deployment for their particular use case.

Conclusion

As this is a new product VMware is keen on collecting feedback from vCloud Director service providers on which additional storage platforms and new features should be added in the next version. You can engage with the product team via the VMware Communities website.