Understanding Datastore Metrics in vCloud Director

I got question from a customer what is the meaning of the various datastore metrics in vCloud Director: used, provisioned and requested storage. These can be viewed at the Datastores & Datastores Cluster screen in the Manager & Monitor tab.

Datastore list

There are even more metrics when we open properties of a single datastore or a datastore cluster.

Datastore Properties

So let us go through them:

Total Capacity: Total size of the datastore or datastore cluster as reported by vCenter Server

Used Capacity: Actual used capacity of the datastore or the datastore cluster as reported by vCenter Server.

Available Capacity: Difference between the values above (Available Capacity = Total Capacity – Used Capacity)

I am including screenshot from vCenter Server of the same datastore:

Datastore Properties in vCenter

Provisioned Capacity: Total storage provisioned of virtual machines residing on the datastore. This number might be much bigger than the actual datastore capacity if thin provisioning is used. Again, this number is reported by vCenter Server (can be seen in the Datastore and Datastore Clusters view on the Summary tab). I am again including relevant vCenter Server screenshot.

vCenter Provisioned

Requested Storage: This is the only metric coming directly from vCloud Director. It adds up storage for all vCloud Director provisioned virtual machines, catalog templates and media and vCNS (vShield) Edges. It uses allocated storage and also includes (even potential) memory swap for virtual machines (not for catalog VMs). It does not include storage occupied by shadow VMs or intermediate disks in a linked clone tree.

Fast Provisioning Enhancement in vCloud Director 5.5

Fast provisioning was introduced with vCloud Director 1.5 and enables speeding up a cloning process when deploying vApps from catalog or copying VMs. It utilizes vSphere linked clones where the base image is not cloned, instead a delta disk is created to record changed blocks.

vCloud Director 5.1 enabled fast provisioning support of VAAI accelerated hardware clones with NFS arrays that offered this feature. I described in detail how the feature works with NetApp flex clones here. Where linked clones bring performance impact on reads due to the need of traversing the chain of delta disks to find the right block and also due to problem that the delta disk is not properly aligned, hardware clones (depending on storage array implementation) might have no such performance impact.

One of the main characteristic of linked clones (vSphere native or hardware accelerated) is the need to have the base disk on the same datastore as the delta disks. vCloud Director placement engine always prefers to place the fast provisioned clone on the datastore with base image, however there might be situations when it is not possible. The datastore might be full (red threshold), the target VM should be placed to different storage profile (renamed to storage policy in 5.5), to a cluster that does not have access to the base image catalog or to a different vCenter Server. In such cases if the original VM was a catalog template, a shadow VM was fully cloned on the target datastore first and then linked clone was created.


The problem with this approach is that where linked clone creation takes seconds, shadow VM creation could take minutes. This leads to inconsistent quality of service to the end-users. I have always recommended to service providers to ‘pre-warm’ datastores by manually forcing creation of shadow VMs for their catalog images. The good news is that now in vCloud Director 5.5 there is a hidden advanced config which will do this for you automatically.

Following entry must be added to the config table in vCloud Director database: valc.catalog.fastProvisioning=true. This can be accomplished with this SQL statement:

INSERT into [vcloud].[dbo].[config] (cat,name,value)
VALUES (‘vcloud’,’valc.catalog.fastProvisioning’,’true’);

From now on vCloud Director will once a day (every 24 hours from its first start) check status of all catalog templates and will proactively create shadow VMs for each combination of storage policy and cluster. It will make sure that there is at least one datastore with enough space (below yellow threshold) with the shadow VM in each possible storage policy and cluster combination so any provisioning operation will be very fast.

Shadow VMs created on two additional datastores which are members of two different storage policies
Shadow VMs created on two additional datastores which are members of two different storage policies

Some additional notes:

  • It is all or nothing feature. It is not possible to selectively enable only for certain templates.
  • To use this feature, database editing is required. Generally unless a database edit is documented in official VMware source (documentation, KB or support request) such action should be considered as unsupported.

Datastore Migrations in vCloud Director – Part 3 of 3: Datastore Clusters

This article is the last in the series.

Part 1: Introduction
Part 2: Individual Datastores
Part 3: Datastore Clusters

In this scenario we are exposing to vCloud Director only datastore clusters. vCloud Director cannot disable individual datastores, on the other hand we could leverage SDRS Maintenance Mode. We can also break the cluster and remove the old datastore from it and basically be in the Scenario 1 but there is no fun in that.

To leverage SDRS Maintenance Mode we need to add the new datastores to the Datastore Cluster (beware of limit of maximum of 32 datastores per cluster). Note: it is not possible to mix NFS and block based datastores. Obviously Storage DRS must be enabled (manual mode is enough).

1. When we put the old datastore to SDRS Maintenance Mode following will happen: vCloud Director will not deploy any new VMs or store media on it.

2. Storage DRS will either automatically or will wait for confirmation (if in manual mode) migrate all VMs away from the datastore. As of vSphere 5.1 this also supports linked clones (vCloud Director Fast Provisioning). Meaning that it will not inflate a linked clone, instead it will try to first migrate the link clone to a datastore with shadow VM (copy of the base image). If there is none, it will first copy the base image and then create linked clone. This has some limitation though. If we are migrating VM2 and VM3 which are linked clones of VM1 at the same time we will have two times full clone of VM1 and then from each linked clone VM2 and VM3 respectively. Therefore I would advise to limit number of concurrent Storage vMotion operations per datastore – see here.

All regular VMs, catalog VMs, expired VMs and vShield Edges will be migrated.

Note: If VAAI accelerated Fast Provisioning is used (hardware snapshots) this will not work!!! As mentioned in Part 1 vSphere 5.1 does not support vMotion of hardware clones.

3. Catalog media files will not be migrated as they are not vSphere objects. Here we need to use the same procedure as mentioned in the Scenario 1, step 3.

In my opinion there is strong case for using Datastore Clusters with vCloud Director to simplify vCloud storage management. The important consideration is on which level we want to do the management – vSphere of vCloud? Depending on the internal politics and separation of duties this might be the decisive factor.

Datastore Migrations in vCloud Director – Part 2 of 3: Individual Datastores

This article is 2nd part of the series.

Part 1: Introduction
Part 2: Individual Datastores
Part 3: Datastore Clusters

In this scenario we are presenting to vCloud Director individual datastores, not datastore clusters. New datastores were provisioned with the same Storage Capability and to the same hosts as the old ones which we want to evacuate and subsequently remove both from vCloud Director and vSphere.

1. Disable the old datastores in vCloud Director (System > Manage and Monitor > vSphere Resources > Datastores & Datastore Clusters).

2a. To relocate a VM its storage profile needs to be updated. Even if the storage profile remains the same when it is updated vCloud Director will verify that current VM location complies with given storage profile. If it is on disabled datastore it will move it to a different datastore of the same storage profile. This can be accomplished either from GUI or with vCloud API.

a. GUI – deployed VM: Open VM Properties, scroll down, change the Storage Profile to another and then back again. This will activate the OK button. Obviously we need at least two Storage Profile assigned to the Org VDC.
VM Storage Profile

b. GUI – VM in catalog: The process is almost the same. Open the VM properties in the catalog and assign the default storage profile and click OK button.
Catalog VM Storage Profile

c. GUI – Expired VM: For expired regular or catalog VM it is not possible to update storage profile from within the GUI.

d. API: The process for regular, catalog or expired VMs is identical. Retrieve the Vm element with GET request and then update the Vm with PUT request with identical body. http://pubs.vmware.com/vcd-51/topic/com.vmware.vcloud.api.doc_51/GUID-E56EEEEA-EDED-4EBC-8095-08AD2668F0D2.html

2b. As an alternative to the relocation initiated from vCloud Director we could accomplish the same with Storage vMotion at the vSphere layer. However there are same caveats:

  • Make sure that the new datastores have the correct Storage Capability and are assigned to the same hosts. There is no vCloud Director placement engine verification!
  • As stated in the Part 1 this is not recommended for Fast Provisioned VMs as they will be inflated to full clones
  • As stated in Part 1 this is not possible for VAAI Fast Provisioned VM as vSphere cannot storage vMotion them
  • You will need to manually load balance the VMs based on their size and available disk space of the new datastores

3. Other objects that can be stored on datastore we need to evacuate are vCloud Director catalog media (ISO and FLP files). These files are stored in special directory with following name structure: <VCD Instance Name>/media/<org ID>/<org VDC ID>/media-<media ID>.<ISO|FLP>.

a. GUI: In the catalog find the media object, right click and select Move to catalog. Pick the same VDC it resides in currently and select storage profile: Same As Source. Note: you will see vCenter Copy task right away; the deletion of the source media does not happen immediately upon the file copy completion, but can take a few minutes as it is asynchronous cleanup operation.
Catalog Media

b. API: The process is identical to the Storage Profile update of a VM. Retrieve the Media element with GET request and update it with PUT request with identical body.
There is no other way to move media files. Manual file transfer will make the media unavailable to vCloud Director as it references the target datastore in its database.

4. Last vCloud Director objects stored on to be evacuated datastore might be vShield Edges. The current (VCD 5.1.2) behavior is that Edge can be deployed to any datastore of the Provider VDC the Org VDC resides in. This is not very intuitive. On top of that it can end up even on disabled datastore, which is a bug and is going to be fixed in the next bug fix release. This means that redeploy from vCloud Director does not help. We need to either Storage vMotion it from vSphere to the new datastore or redeploy it from vShield Manager while specifying the datastore placement for the Edge appliance.


The Part 3 of the series will discuss if usage of Datastore Clusters and Storage DRS might simplify the process.

Go to part 3.

Datastore Migrations in vCloud Director – Part 1 of 3: Introduction

Due to its length the article has been split into three parts:

Part 1: Introduction
Part 2: Individual Datastores
Part 3: Datastore Clusters

In cloud environments the end-user is completely abstracted from the actual physical hardware which hosts cloud workloads. Therefore it is expected that any hardware replacement due to its lifecycle will have minimal impact on the workloads. I have already written an article how to non-disruptively migrate the compute part – virtual datacenter from old servers to new ones: vCloud Director: Online Migration of Virtual Data Center. This series of articles talks about storage migration.

Although there is a KB article 2058094 about the subject it is not comprehensive and some important considerations are not mentioned.

When a datastore or multiple datastores are decommissioned, vApp virtual machines, catalog vApps, catalog media and vShield Edges must be migrated away. The process depends if we are migrating to a new storage profile or not, if we use fast provisioning or storage clusters.

vCloud Director 5.x consumes vSphere datastores through storage profiles which are assigned to Provider VDCs. Individual datastores cannot be added or removed from a Provider VDC directly. vCloud Director placement engine is choosing datastores for placement based on the storage profile and host connectivity. If datastore cluster with Storage DRS is used then vCloud Director placement engine relies on Storage DRS placement recommendations. Datastore cluster must have all datastore with the same Storage Capability and it is not possible to mix NFS with block based datastores. It is not possible to manage individual datastores of a datastore cluster in vCloud Director – meaning disabling them, setting disk space thresholds or enabling VAAI for fast provisioning. Those settings now apply to the whole datastore cluster. Speaking of fast provisioning, Storage vMotion of a linked clone produces full clone, storage vMotion of VAAI accelerated linked clone (hardware snapshot) is not possible (as of vSphere 5.1).

Understanding these facts it should be now easier to derive the right migration steps.

In the next article I will talk how to non-disruptively  migrate vCloud Director infrastructures from storage that is presented as individual datastores. The last article will talk about usage of datastore clusters and Storage DRS with vCloud Director.

Go to Part 2.