Category Archives: Storage

Datastore Cluster Issue in vCloud Director 5.5.1

Just found out there is a known issue with removing datastores in and out of datastore clusters. In vCloud Director 5.1 this was working fine. vCloud Director refers to storage via storage policies (formerly profiles) so you could on-the fly change the datastore cluster structure (as long as all datastore inside had the same storage policy).

However in vCloud Director 5.5.1 if you move a datastore in or out of a datastore cluster, vCloud Director will loose it. The fix is described in KB 2075366 and involves clean up of vCloud Director database inventory data.

Monitor Wear and Tear of Your VSAN SSDs

As I started experimenting with VMware Virtual SAN (VSAN) in my lab and am using consumer grade SSDs which are not on VSAN (beta) HCL I am worried about wear and tear of the memory cells who have limited write endurance.

I wondered if it is possible to access the SMART attributes of the disks and quick search showed that there is a KB article 2040405 written which still applies although not specific to vSphere 5.5.

From the ESXi console run esxcli storage core device list to get list of storage devices and then run esxcli storage core device smart get -d device to get the SMART data.

This is output of my Intel 520 drive.

Intel 520 SMART Attributes


Unfortunately another host with OCZ SSD does not display any data with an error:

Error getting Smart Parameters: CANNOT open device


Understanding Datastore Metrics in vCloud Director

I got question from a customer what is the meaning of the various datastore metrics in vCloud Director: used, provisioned and requested storage. These can be viewed at the Datastores & Datastores Cluster screen in the Manager & Monitor tab.

Datastore list

There are even more metrics when we open properties of a single datastore or a datastore cluster.

Datastore Properties

So let us go through them:

Total Capacity: Total size of the datastore or datastore cluster as reported by vCenter Server

Used Capacity: Actual used capacity of the datastore or the datastore cluster as reported by vCenter Server.

Available Capacity: Difference between the values above (Available Capacity = Total Capacity – Used Capacity)

I am including screenshot from vCenter Server of the same datastore:

Datastore Properties in vCenter

Provisioned Capacity: Total storage provisioned of virtual machines residing on the datastore. This number might be much bigger than the actual datastore capacity if thin provisioning is used. Again, this number is reported by vCenter Server (can be seen in the Datastore and Datastore Clusters view on the Summary tab). I am again including relevant vCenter Server screenshot.

vCenter Provisioned

Requested Storage: This is the only metric coming directly from vCloud Director. It adds up storage for all vCloud Director provisioned virtual machines, catalog templates and media and vCNS (vShield) Edges. It uses allocated storage and also includes (even potential) memory swap for virtual machines (not for catalog VMs). It does not include storage occupied by shadow VMs or intermediate disks in a linked clone tree.

Fast Provisioning Enhancement in vCloud Director 5.5

Fast provisioning was introduced with vCloud Director 1.5 and enables speeding up a cloning process when deploying vApps from catalog or copying VMs. It utilizes vSphere linked clones where the base image is not cloned, instead a delta disk is created to record changed blocks.

vCloud Director 5.1 enabled fast provisioning support of VAAI accelerated hardware clones with NFS arrays that offered this feature. I described in detail how the feature works with NetApp flex clones here. Where linked clones bring performance impact on reads due to the need of traversing the chain of delta disks to find the right block and also due to problem that the delta disk is not properly aligned, hardware clones (depending on storage array implementation) might have no such performance impact.

One of the main characteristic of linked clones (vSphere native or hardware accelerated) is the need to have the base disk on the same datastore as the delta disks. vCloud Director placement engine always prefers to place the fast provisioned clone on the datastore with base image, however there might be situations when it is not possible. The datastore might be full (red threshold), the target VM should be placed to different storage profile (renamed to storage policy in 5.5), to a cluster that does not have access to the base image catalog or to a different vCenter Server. In such cases if the original VM was a catalog template, a shadow VM was fully cloned on the target datastore first and then linked clone was created.


The problem with this approach is that where linked clone creation takes seconds, shadow VM creation could take minutes. This leads to inconsistent quality of service to the end-users. I have always recommended to service providers to ‘pre-warm’ datastores by manually forcing creation of shadow VMs for their catalog images. The good news is that now in vCloud Director 5.5 there is a hidden advanced config which will do this for you automatically.

Following entry must be added to the config table in vCloud Director database: valc.catalog.fastProvisioning=true. This can be accomplished with this SQL statement:

INSERT into [vcloud].[dbo].[config] (cat,name,value)
VALUES (‘vcloud’,’valc.catalog.fastProvisioning’,’true’);

From now on vCloud Director will once a day (every 24 hours from its first start) check status of all catalog templates and will proactively create shadow VMs for each combination of storage policy and cluster. It will make sure that there is at least one datastore with enough space (below yellow threshold) with the shadow VM in each possible storage policy and cluster combination so any provisioning operation will be very fast.

Shadow VMs created on two additional datastores which are members of two different storage policies

Shadow VMs created on two additional datastores which are members of two different storage policies

Some additional notes:

  • It is all or nothing feature. It is not possible to selectively enable only for certain templates.
  • To use this feature, database editing is required. Generally unless a database edit is documented in official VMware source (documentation, KB or support request) such action should be considered as unsupported.

Datastore Migrations in vCloud Director – Part 3 of 3: Datastore Clusters

This article is the last in the series.

Part 1: Introduction
Part 2: Individual Datastores
Part 3: Datastore Clusters

In this scenario we are exposing to vCloud Director only datastore clusters. vCloud Director cannot disable individual datastores, on the other hand we could leverage SDRS Maintenance Mode. We can also break the cluster and remove the old datastore from it and basically be in the Scenario 1 but there is no fun in that.

To leverage SDRS Maintenance Mode we need to add the new datastores to the Datastore Cluster (beware of limit of maximum of 32 datastores per cluster). Note: it is not possible to mix NFS and block based datastores. Obviously Storage DRS must be enabled (manual mode is enough).

1. When we put the old datastore to SDRS Maintenance Mode following will happen: vCloud Director will not deploy any new VMs or store media on it.

2. Storage DRS will either automatically or will wait for confirmation (if in manual mode) migrate all VMs away from the datastore. As of vSphere 5.1 this also supports linked clones (vCloud Director Fast Provisioning). Meaning that it will not inflate a linked clone, instead it will try to first migrate the link clone to a datastore with shadow VM (copy of the base image). If there is none, it will first copy the base image and then create linked clone. This has some limitation though. If we are migrating VM2 and VM3 which are linked clones of VM1 at the same time we will have two times full clone of VM1 and then from each linked clone VM2 and VM3 respectively. Therefore I would advise to limit number of concurrent Storage vMotion operations per datastore – see here.

All regular VMs, catalog VMs, expired VMs and vShield Edges will be migrated.

Note: If VAAI accelerated Fast Provisioning is used (hardware snapshots) this will not work!!! As mentioned in Part 1 vSphere 5.1 does not support vMotion of hardware clones.

3. Catalog media files will not be migrated as they are not vSphere objects. Here we need to use the same procedure as mentioned in the Scenario 1, step 3.

In my opinion there is strong case for using Datastore Clusters with vCloud Director to simplify vCloud storage management. The important consideration is on which level we want to do the management – vSphere of vCloud? Depending on the internal politics and separation of duties this might be the decisive factor.