Just a quick announcement that a vCloud Director Object Storage Extension Reference Design that I wrote is now available at this link.
It deep dives into the use cases, architecture, includes recommended deployment options and description of the new feature of 1.0.1 related to multisite deployments. There are also results of performance tests of the overhead the extension adds over the direct use of the native storage platform.
VMware released last week another product that extends vCloud Director and enables Cloud Service Providers to offer additional services on top of vCloud Director out-of-the-box IaaS. Where vCloud Availability adds Disaster Recovery and migration services to vCloud Director, Container Service Extension adds the ability to deploy Kubernetes clusters, vRealize Operations Tenant App brings advanced workload monitoring, the newly released vCloud Director Object Storage Extension offers easy access for the tenants to a scalable, cheap, durable and network accessible storage for their applications.
As the name suggests it is an extension, that lives side by side to vCloud Director and that requires 3rd party object storage provider. In the 1.0 release the only supported storage provider is Cloudian Hyperstore, however other storage providers (cloud or on-prem) are coming in future releases. The extension provides multitenant S3 compatible API endpoint as well as user interface plugin for vCloud Director.
The object storage service is fully in the service provider competence who decides its parameters (SLAs, scalability) and upsells it to existing or new vCloud Director tenants.
The tenants can provision storage buckets and directly upload/download objects into them via the UI, or use S3 APIs or S3 compatible solutions to do so. Objects can be also accessible via S3 path-style URL for easy sharing.
Additionally tenants can provision application credentials and use them in their (stateless) workloads to persist application configuration or logs and have access to unstructured data (web servers).
Tight integration with vCloud Director also offers usage of object storage as archival or distribution resource for vCloud Director vApps and Catalogs. Tenant can capture existing vApps to a dedicated object storage bucket and later restore it to its Org VDCs.
Alternatively whole vCloud Director Organization Catalog of vApp templates and ISO images can be captured to the bucket or created from scratch by uploading individual ISO and OVA objects and used by same or another Organization even in a different vCloud Director instance via the catalog subscribe mechanism.
The solution supports S3 API with AWS Signature V4, which means existing applications can easily leverage the Object Storage service without the need for rewrites. The below screenshots show usage of S3 Browser freeware Windows client to manage the files.
Objects can be tagged and assigned with metadata, buckets can be tagged as well. Server side encryption can be configured by the Org Admin at tenant level or via API at object level. SSE-S3 (server managed key) and SSE-C (client supplied keys) methods are supported. Access Control List (ACL) permissions can be set at bucket or at object. Buckets can be shared within the tenant (to subset or all users) or made public.
Security credentials (pair of access and secret keys) are of two types. User credentials (can manage all users buckets and objects) and application credentials (can only manage subset of buckets). Object Storage Extension automatically creates user credential for each tenant user, however additional user or application credentials can be created. Credentials can be disabled and/or deleted.
The full set of supported S3 APIs is documented via the swagger UI on the extension endpoint (/docs) or here.
While the object storage tenant consumption APIs are standardized (S3 AWS APIs), each storage platform uses different admin APIs. Object Storage Extension currently does not expose provider APIs. The tenant administration (service entitlement) is done from the vCloud Director provider UI.
Other administration (quotas, usage metering, platform monitoring, etc.) are done directly through the Cloudian Management Console where the provider admin is redirected from the vCloud Director UI or optionally through Cloudian HyperStore Admin APIs.. This will change in later releases when more storage providers are supported.
Object Storage Service uses three different user personas. Provider administrator, tenant administrator and tenant user. Provider administrator manages tenant access to service and the storage platform. Tenant administrator has access to all buckets and objects of a particular tenant and can monitor consumption at organization, user or bucket level. Tenant user can only access her own buckets and objects or the ones shared with the user.
The user personas map to users based on their vCloud Director rights. The mapping in general corresponds to System Administrator / Organization Administrator / other non Organization Administrator global roles, unless these were changed in vCloud Director.
Provider Administrator (system context):
Tenant Administrator:
Tenant User:
The Object Storage Extension has 1:1 relationship with vCloud Director instance and 1:1 relationship with the storage provider (Cloudian HyperStore). Each vCloud Director Organization that is enabled to consume the service will have unique counterpart at the storage platform (Cloudian HyperStore business groups). Same is valid for users. As it is vCloud Director who provides authentication to the service, it is fully multitenant.
The diagram (taken from the official documentation) below shows all the components needed for the Object Service Extension including the traffic flows. vCloud Director 9.1 and newer is supported. Next to the vCloud Director cells you will need to deploy one or more (for HA and scalability) RHEL/CentOS/Oracle Linux VM nodes (dark green in the picture) that will run the Object Storage Extension service that is provided is RPM package. These VMs are essentially stateless and persist all their data in PostgreSQL DB. This could be vCloud Director external PostgreSQL DB (if possible) or a dedicated database just for the Object Storage Extension.
The service needs its own public IP address as it runs (by default) on port 443. S3 API clients or the vCloud Director UI plugin will access this endpoint. vCloud API extensibility is not used, but vCloud Director HTML 5 UI extensibility is.
The extension VM nodes need to have access to vCloud API endpoint for user authentication and for the vApp/Catalog import/export functionality. Additionally they will need fast access to the underlying object storage platform (in our case Cloudian HyperStore). Cloudian HyperStore is fully distributed with a minimum supported deployment of three (fully equivalent) storage nodes and scales essentially indefinitely. Each storage node also provides UI/API functionality. Fast L4 load balancing should be used to forward the extension calls to all storage nodes. Multiple APIs (S3, IAM and Admin) each running on separate TCP port need to be accessed as well as Cloudian Management Console for the Provider UI plugin redirection (this is the only service that needs to be set up with sticky sessions).
As can be seen the Object Storage Extension is in the datapath of the object transfers that are persisted on the storage nodes. The overhead is less than 10% when compared to accessing Cloudian directly (with TLS sessions) however the extension nodes must be sized properly (it is a CPU intensive workload) so they do not become a bottleneck. Both scale-out and scale-up options are possible.
The Cloudian HyperStore storage nodes can be deployed in three different configurations. For small environments or testing it can be deployed as virtual appliance running on vSphere (CentOS + HyperStore binary) leveraging shared (more expensive) or local disk storage (HyperStore replicates objects across storage nodes so it does not need highly available shared storage). Another options are to deploy Cloudian Hyperstore on dedicated bare metal hardware or to purchase hardware appliances directly from Cloudian. It is up to service provider to decide which form factor to use to tailor the deployment for their particular use case.
As this is a new product VMware is keen on collecting feedback from vCloud Director service providers on which additional storage platforms and new features should be added in the next version. You can engage with the product team via the VMware Communities website.
Tom Fojta's Blog 