Although cloud services are providing access to abstracted seemingly infinite physical resources, the truth is that the physical infrastructure is not limitless. Pooling and distributed resource scheduling for compute, storage and network helps but at the end there is always a physical host, LUN or network uplink which constraints the granularity of scaling.
When it comes to storage it is the datastore size that limits the maximum size of virtual disk a cloud consumer can attach to his/her VM. While thin and fast provisioning and dedupe (NFS/VSAN) can be used to fit more data and storage DRS can shuffle the data around when a particular datastore is filling up at the end the service provider should not allow creation of arbitrary size of vdisks (vSphere maximum is 62 TB) to avoid datastore out of space condition. For example letting customers provision 4 TB thin disks on 3 TB LUNs is just asking for trouble.
Before vCloud Director 8.10 service providers were leveraging blocking tasks with custom orchestration to check if provisioned VM is within provider specified limits (RAM size, vDisk size, max vCPUs). There is reference implementation published here: CPU and Memory Limit enforcement for vCloud Director.
vCloud Director 8.10 brings hidden configuration option where service provider can globally set the maximum allowed size of virtual disk.
The option can be set with cell-management-tool command on a vCloud cell with the following syntax:
$VCLOUD_HOME/bin/cell-management-tool manage-config -n vmlimits.disk.capacity.maxMb -v 1000000
which would set maximum size of disk to 1000000 MB which is 1 TB.
Note: the command is run on one vCloud cell and its impact is immediate (no need to restart anything).
If the tenant tries to provision larger vDisk he will get the following error:
Note that the limit is not enforced for system administrators and existing disks are not affected.
What should be the limit is out of scope for this post as there are many considerations that should be taken into account:
- datastore size
- can datastore grow?
- thin provisioning
- fast provisioning
- tenant snapshots
- provider snapshots (backup software generated)
- yellow and red datastore thresholds
- Storage DRS
- deduplication on the array
Just found out there is a known issue with removing datastores in and out of datastore clusters. In vCloud Director 5.1 this was working fine. vCloud Director refers to storage via storage policies (formerly profiles) so you could on-the fly change the datastore cluster structure (as long as all datastore inside had the same storage policy).
However in vCloud Director 5.5.1 if you move a datastore in or out of a datastore cluster, vCloud Director will loose it. The fix is described in KB 2075366 and involves clean up of vCloud Director database inventory data.
As I started experimenting with VMware Virtual SAN (VSAN) in my lab and am using consumer grade SSDs which are not on VSAN (beta) HCL I am worried about wear and tear of the memory cells who have limited write endurance.
I wondered if it is possible to access the SMART attributes of the disks and quick search showed that there is a KB article 2040405 written which still applies although not specific to vSphere 5.5.
From the ESXi console run esxcli storage core device list to get list of storage devices and then run esxcli storage core device smart get -d device to get the SMART data.
This is output of my Intel 520 drive.
Unfortunately another host with OCZ SSD does not display any data with an error:
Error getting Smart Parameters: CANNOT open device
I got question from a customer what is the meaning of the various datastore metrics in vCloud Director: used, provisioned and requested storage. These can be viewed at the Datastores & Datastores Cluster screen in the Manager & Monitor tab.
There are even more metrics when we open properties of a single datastore or a datastore cluster.
So let us go through them:
Total Capacity: Total size of the datastore or datastore cluster as reported by vCenter Server
Used Capacity: Actual used capacity of the datastore or the datastore cluster as reported by vCenter Server.
Available Capacity: Difference between the values above (Available Capacity = Total Capacity – Used Capacity)
I am including screenshot from vCenter Server of the same datastore:
Provisioned Capacity: Total storage provisioned of virtual machines residing on the datastore. This number might be much bigger than the actual datastore capacity if thin provisioning is used. Again, this number is reported by vCenter Server (can be seen in the Datastore and Datastore Clusters view on the Summary tab). I am again including relevant vCenter Server screenshot.
Requested Storage: This is the only metric coming directly from vCloud Director. It adds up storage for all vCloud Director provisioned virtual machines, catalog templates and media and vCNS (vShield) Edges. It uses allocated storage and also includes (even potential) memory swap for virtual machines (not for catalog VMs). It does not include storage occupied by shadow VMs or intermediate disks in a linked clone tree.
Fast provisioning was introduced with vCloud Director 1.5 and enables speeding up a cloning process when deploying vApps from catalog or copying VMs. It utilizes vSphere linked clones where the base image is not cloned, instead a delta disk is created to record changed blocks.
vCloud Director 5.1 enabled fast provisioning support of VAAI accelerated hardware clones with NFS arrays that offered this feature. I described in detail how the feature works with NetApp flex clones here. Where linked clones bring performance impact on reads due to the need of traversing the chain of delta disks to find the right block and also due to problem that the delta disk is not properly aligned, hardware clones (depending on storage array implementation) might have no such performance impact.
One of the main characteristic of linked clones (vSphere native or hardware accelerated) is the need to have the base disk on the same datastore as the delta disks. vCloud Director placement engine always prefers to place the fast provisioned clone on the datastore with base image, however there might be situations when it is not possible. The datastore might be full (red threshold), the target VM should be placed to different storage profile (renamed to storage policy in 5.5), to a cluster that does not have access to the base image catalog or to a different vCenter Server. In such cases if the original VM was a catalog template, a shadow VM was fully cloned on the target datastore first and then linked clone was created.
The problem with this approach is that where linked clone creation takes seconds, shadow VM creation could take minutes. This leads to inconsistent quality of service to the end-users. I have always recommended to service providers to ‘pre-warm’ datastores by manually forcing creation of shadow VMs for their catalog images. The good news is that now in vCloud Director 5.5 there is a hidden advanced config which will do this for you automatically.
Following entry must be added to the config table in vCloud Director database: valc.catalog.fastProvisioning=true. This can be accomplished with this SQL statement:
INSERT into [vcloud].[dbo].[config] (cat,name,value)
From now on vCloud Director will once a day (every 24 hours from its first start) check status of all catalog templates and will proactively create shadow VMs for each combination of storage policy and cluster. It will make sure that there is at least one datastore with enough space (below yellow threshold) with the shadow VM in each possible storage policy and cluster combination so any provisioning operation will be very fast.
Shadow VMs created on two additional datastores which are members of two different storage policies
Some additional notes:
- It is all or nothing feature. It is not possible to selectively enable only for certain templates.
- To use this feature, database editing is required. Generally unless a database edit is documented in official VMware source (documentation, KB or support request) such action should be considered as unsupported.