They used to be fierce competitors but now are good buddies. Who? vCloud Automation Center (vCAC) and vCloud Director. vCAC version 5.2 has just been released and it is the second release of what was before known as DynamicOps Cloud Automation Center but since the VMware acquisition of DynamicOps in July 2012 has been rebranded under the vCloud brand.

Why would you integrate these two products if they were competing with each other in the past and are perceived to do the same things?

This marchitecture slide below was used by Pat Gelsinger during his EMC World keynote and perfectly shows the relationship of the two.

vCAC is part of the Cloud Service Provisioning pillar and is a tool that can operate above heterogeneous set of infrastructure resources be it vSphere, other hypervisors, public or private clouds and physical servers. vCAC sits (among others) on top of vCloud Director which can be either private or public cloud and provides policy based (that’s the piece that controls What, Who, Where, Why, How much, How long, …) provisioning with nice, simple and extensible GUI.

The previous December 2012 vCAC 5.1 release had very little integration with vCloud Director, required vSphere access and therefore did not work with public clouds. That is not the case anymore and vCloud Director is treated as first class cloud citizen together with Amazon EC2.

Configuration

Here follows brief description how to set up a vCloud Director based cloud as an endpoint and how to create a provisioning blue print.

1. Create vCloud Director endpoint. vCAC Administrator > Endpoints > New Endpoint > Cloud > vApp (vCloud Director). In the public cloud scenario we would use Organization Administrator credentials, in private cloud we could use vCloud Administrator credentials.
   
2. Perform Data Collection of the newly created endpoint. One of the Distributed Execution Manager (DEM) Workers will by using vCloud API connect to the cloud and collect available inventory.
   
3. Once the data collection is finished we can create a new enterprise group which is basically a logical separation of the infrastructure resources. vCAC Administrator > Enterprise Group > New Enterprise Group. Name the group, add the account of the enterprise administrator and select the vCloud resources (OrgVDCs) that will belong here.
   
4. Now we can log in as the Enterprise Administrator and start managing the group. We have to create a provisioning group which is basically a set of users that will be able to provision VMs. Enterprise Administrator > Provisioning Group > New Provisioning Group. I called my provisioning group TestDev
5. Now we can create reservation of resources for the provisioning group.  Enterprise Administrator > Reservations > New Reservation > Cloud > vApp (vCloud Director)
   
   I have assigned 2606-Public OrgVDC to the TestDev provisioning group. In the Resources subtab we can select storage tiers and networks that will be available to this reservation and optionally limit memory and storage. Network profile (set of IP addresses) can be assigned to the networks as well.
   
6. The Who and Where is ready. Now we need to prepare the What. We will create (global) blueprint which will define the VMs that the users can provision. vCloud vApp blueprint consists of component blueprint which defines the actual VMs and a vApp blueprint that specifies policies for the whole vApp.
   So starting with the component blueprint: Enterprise Administrator > Global Blueprints > New Blueprint > Cloud > vApp Component (vCloud Director). In the Blueprint Information tab we assign provisioning groups that can use the blueprint, prefix for the naming of the VMs, optionally approval policy and costs.
   In the Build Information tab we specify the VM template and maximums for its configuration. 
7. Now we can create vApp blueprint: Enterprise Administrator > Global Blueprints > New Blueprint > Cloud > vApp (vCloud Director)
   In the Build Information tab we link the template to the component blueprint.
   
8. Now we can log in as a user from the Provisioning Group, Go to Self-Service and Request a machine from the blueprint.
   
9. After while the machine is ready to be consumed.
   

User Mapping

vCAC provisions and manages the vCloud Director vApps with the administrator account configured in the Cloud Endpoint. However a vApp is created it changes its ownership to the user who requested it. If the vCloud Organization does not contain user with an identical username as the vCAC VM requestor it will try to import the user from LDAP. This can obviously work only if LDAP is configured for the vCloud Organization which is realistic only in private vCloud Director deployments. In public vClouds you will therefore have to make sure that the user (either local or SAML imported) exists in the organization prior to vCAC provisioning.

Licensing

There is one difference when licensing private or public clouds. For private clouds it is possible to use CPU socket based licensing of vCloud Suite Enterprise. That obviously does not work with public clouds and therefore per VM licensing is needed.

The problem statement is following: I have my own Enterprise Certificate Authority which is not publicly trusted and I have issued certificates for my vCC Server and all Nodes without any intermediate certificate. How do I import them?

The vCC VAMI GUI supoprts only import of certificate chain that consist of root>intermediate>certificate. As I do not have intermediate certificate I have to use JAVA keytool command. There are two keystores: cacerts for trusted root certificates and tcserver.jks for the other certificates. My Enterprise CA root certificate must be imported into the former one and the vCC Server/Node certificates to the latter.

Here is the exact procedure:

vCloud Connector Server

1. Log in into the vCC Server console as root and delete the self signed hcserver certificate from tcserver.jks keystore. Note the keystore password (changeme) is hardcoded – do not try to change it.

/usr/java/latest/bin/keytool -delete -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

2. Create new hcserver certificate, where CN (first and last name) is the vCC Server FQDN. Use the same password for the certificate as the keystore password (just hit RETURN when asked).

/usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

What is your first and last name?
 [Unknown]:  vccserver.fojta.com
What is the name of your organizational unit?
  [Unknown]:vCloud Connector Server
What is the name of your organization?
  [Unknown]:  fojta.com
What is the name of your City or Locality?
  [Unknown]:  Prague
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:  CZ
Is CN=vccserver.fojta.com, OU= vCloud Connector Server, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
  [no]:  yes
 Enter key password for <hcserver>
        (RETURN if same as keystore password):

3. Now we have to create the Certificate Signing Request. We can do it either from command line or from the VAMI GUI (Server>SSL>Generate and download CSR)

/usr/java/latest/bin/keytool -certreq -alias hcserver -file hcserver.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

4. Sign the certificate signing request (hcserver.csr) with your Enterprise CA. I have used the Subordinate Certification Authority certificate template. Upload the signed certificate to the vCC server as file hcserver.cer.

5. Before importing hcserver.cer certificate we need to import the CA root certificate otherwise it would not be able to build trusted chain. Obtain the CA root certificate (in my case named fojta-dc-CA.cer and import to cacerts keystore with following command). The alias must be unique. Note the certificate password (changeit) is hardcoded, do not try to change it (no pun intended).

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Now we can import the hcserver certificate

/usr/java/latest/bin/keytool -import -alias hcserver -file hcserver.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

7. Verify that the import was successful.

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcserver, May 10, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): FC:A3:29:96:0D:CD:E2:04:3D:0F:E9:B6:8B:A0:6F:B8:C8:BF:3E:61

vCloud Connector Node

The process is identical to the vCloud Connector Server certificate replacement with just minor changes. The certificate alias is not hcserver but hcagent and the tcserver.jks keystore location is different. Therefore just briefly:

1. Delete selfsigned hcagent certificate
/usr/java/latest/bin/keytool -delete -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

2. Create new hcagent certificate

/home/admin # /usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

What is your first and last name?
[Unknown]:  vccnode.fojta.com
What is the name of your organizational unit?
[Unknown]:vCloud Connector Node
What is the name of your organization?
[Unknown]:  fojta.com
What is the name of your City or Locality?
[Unknown]:  Prague
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]:  CZ
Is CN=vccnode.fojta.com, OU=vCloud Connector Node, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
[no]:  yes

Enter key password for <hcagent>
(RETURN if same as keystore password):

3. Create CSR or download from VAMI GUI: Node>SSL>Generate and download CSR

/usr/java/latest/bin/keytool -certreq -alias hcagent -file hcagent.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

4. Sign hcagent.csr with your CA => hcagent.cer

5. Import CA root certificate (fojta-dc-CA.cer)

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Import hcagent certificate

/usr/java/latest/bin/keytool -import -alias hcagent -file hcagent.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

7. Verify

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcagent, May 10, 2013, PrivateKeyEntry
Certificate fingerprint (SHA1): 9E:63:B1:BC:91:FE:7D:84:FC:8C:66:24:B9:1B:B9:73:80:5D:AC:87

Enabling SSL

Now we should be able both on vCC Server and all vCC Nodes to enable SSL (in Server/Node tab, SSL subtab click Enable SSL button). When returning the the SSL tab it seems vCC is not able to get the current SSL status correctly. vCC Server displays an error and vCC Node nothing – not really sure why is this happening, just ignore it.

Now we should be able in the Nodes tab on the vCC Server edit the Node configuration and uncheck Ignore SSL Certificate. If the certificate replacement was successful, the Node Status shoud be Up.

For more details rather refer to the online vCloud Connector documentation as it contains more information than the PDF docs.

One of the new features of vCloud Director v 5.1 was elastic Allocation pool VDC. Elastic means that the VDC can span multiple clusters which simplifies providers capacity management.

The feature required some changes how now elastic VDC maps to vSphere Resource Pools. And these changes were disruptive for some customers upgrading from vCloud Director 1.5. Therefore both vCloud Director 5.1.1 and 5.1.2 tweaked the feature to make those customer happy.

For deep dive how Org VDC allocation types relate to vSphere resource management go to Massimo Re Ferre post here: vCloud Director 5.1(.1) Changes in Resource Entitlements (Updated).

I will just concentrate on the Allocation Pool VDC differencies.

vCloud Director 5.1.0

Allocation pool VDC require new parameter: vCPU speed, which is used to define how much CPU reservation and limit is applied to Org VDC resource pools that can span multiple clusters. Each such resource pool gets reservation and limit based on sum of all vCPUs of deployed vApps in that particular resource pool.

Example: If vCPU parameter is set to 1GHz and I have deployed 3 VMs each with 2 vCPUs and one is placed into one resource pool and the rest to the other, the first resource pool will get 4 GHz limit and the second 8 GHz (reservation is set as a percentage of the limit).

This means that you cannot overallocate Org VDC in terms of vCPUs (max #of vCPUs x vCPU speed = Org VDC CPU allocation) in very similar way the memory could not be overallocated in vCloud Director 1.5.

vCloud Director 5.1.1

As mentioned above some customers complained that the vCloud Director tenants are now constrained in how many vCPUs they can deploy into their Org VDC. Providers tried to fight this with setting very small vCPU speeds, but the problem is that if you have only a few VMs deployed the resource pool limit was very low compared to the allocated Org VDC CPU GHz.

vCloud Director 5.1.1 came with a quick fix. The CPU limit of Allocation pool resource pools was no longer based on number of vCPUs deployed in the resource pool as in 5.1.0, but was the whole Org VDC CPU allocation instead. This means that even the first (and only) deployed vCPU can utilize the full Org CPU Allocation (obviously limited by the physical speed of the core). The downside is that if the Org VDC spans multiple resource pools, the tenant will get more CPU resources then he is entitled to. However as long the provider designed all his Provider VDCs to be backed by only one cluster/resource pool and set low vCPU speed the behavior was very similar to vCloud Director 1.5.

vCloud Director 5.1.2

The problem with the previous approach was that if you upgraded to 5.1.1 you could not revert to 5.1.0 with the truly elastic VDCs if you wanted. That has changed now with 5.1.2.

There is a new “Make Allocation pool Org VDCs elastic” configuration option in System Settings > General > Miscellaneous which gives you the possibility to choose the Allocation Pool behavior.

When upgrading from vCloud Director 5.1.0 that used Allocation Pool Org VDC spanning multiple clusters this option will be enabled, otherwise it will always be disabled by default.

If it is disabled then the Allocation Pool Org VDCs behave exactly as in vCloud Director 1.5. That means no vCPU speed setting, no spanning of multiple clusters and easy vCPU overallocation.

If the option is enabled then the Allocation Pool Org VDCs behave exactly as in vCloud Director 5.1.1!  So beware – it does not revert to 5.1.0 way of setting the resource pool CPU limit, but uses the 5.1.1 way which results in possibility that tenant will use more CPU resources than is his Org VDC CPU allocation.

Personally I have hoped that the elastic behavior would be exactly as in 5.1.0 which is not the case, but could happen in the future releases.

I always recommend to upgrade vCloud Director in two phases.

Phase 1 (vCloud Director Upgrade)

• vCloud Director Cell operating system (RHEL). RHEL 5 is still supported but if customer wants to use RHEL 6 he will need to deploy a new cell as RHEL 5 to RHEL 6 upgrade is not possible.
• vCloud Director runtime upgrade
• vCloud Director database schema upgrade
• vShield Manager upgrade
• vShield Edges upgrade

Phase 2 (vSphere Upgrade)

• Installation of SSO
• Installation of Inventory Service
• Installation/upgrade of Web Client
• vCenter Server upgrade
• ESX hosts upgrade
• distributed virtual switches upgrade

As the phases can be spread out in time this brings the main topic of the article – which new vCloud Director 5.1 features depend on vSphere 5.1 and will not be available during the time between Phase 1 and Phase 2? I have compiled a table which lists the new vCloud Director features and if that feature will be available with vSphere 5.0 (vCenter 5.0 + ESX 5.0. Note: I don’t dare to consider ESX 4).

¹) With vSphere 5.0 vCloud Director does not use SDRS recommendation for linked clone placement (Fast Provisioning). vCloud Director picks individual datastore and optionally deploys shadow VM. With vSphere 5.1 vCloud Director fully leverages SDRS recommendations, shadow VMs are deployed by vSphere SDRS.

Table in PNG format.

Disclaimer: I don’t claim this table is complete and that it is an official VMware document. If you think something is missing, please comment and I will edit the table.

Edit 27 April 2013: Explained difference in linked clone placement.

vCloud Connector (vCC) consists of vCloud Connector Server, vCloud Connector Nodes and vCloud Connector Client. In the past the vCloud Connector was accessible either through vCenter Server plugin or through web interface at http://vcloud.vmware.com/connector. However as of April 2nd 2013 VMware discontinued the web interface access which means that vCenter Server plugin is currently the only option to use vCloud Connector.

The web portal was great for the Developer Use Case. The developer did not need to have access to vSphere, vCenter, vSphere client or even a Windows machine and could use vCloud Connector from his Mac or Linux browser.

What now follows in this article is basically a description how the Developer Use Case can still be fulfilled even after April 2nd. The idea is to use vCenter Server Appliance and deploy it to the cloud together with vCloud Connector Server. This vCenter Server Appliance will not manage any ESX hosts and will be basically used only for the instantiation of the vCloud Connector Client interface. The thick (.NET) vSphere Client is still needed (as at the moment there is no vCC plugin for vSphere Web Client), this also means a Windows OS, so the developer will need either physical Windows desktop, or a virtual one on his PC or running also in the cloud and accessible via RDP.

How to deploy vCenter Server to the public vCloud Director cloud

1. From VMware website download vCenter Server Appliance. I have used the version 5.1.0b which comes as one large OVA file.
2. As we cannot import OVA file to vCloud Director we first need to unzip the file to get the OVF format. This can be done easily by adding .zip extension to the downloaded file and using WinZip or similar utility.
3. Import the OVF file into your organization catalog.
4. As I also had vCC Server and vCC Node in the catalog I deployed them together into one vApp.
   
   
5. After accepting EULA, selecting Storage Profile and setting hostnames it is important to put the VMs on one internet routable network and manually assign IP addresses from the static pool of the organization VDC network. We cannot just rely on Guest Customization as the IP address assignment is part of the vApp property which is applied when the vApp is deployed. So in my case I used (the default) 192.168.1.0/24 subnet for the org VDC network, where IP 192.168.1.1 was used for the internal interface of the Edge Gateway and IPs 192.168.1.2-192.168.1.4 were used for vCenter Server, vCloud Connector Server and vCloud Connector Node.
6. On the next page we are presented with the vApp properties. Here we have to again manually assign the correct IP addresses as specified in the previous step. As I am deploying 3 imported vApps at once it looks quite confusing as the properties of each are merged into one screen. The default gateway address is the Edge Gateway internal interface and subnet is 255.255.255.0.
7. After the deployment is finished we will get vCloud Connector vApp which looks like this:
8. We will need to access all three VMs from the internet to configure their Virtual Appliance Management Interface (VAMI) which runs on TCP port 5480. If you have 3 external IP addresses you can set up destination NAT (DNAT) rules for each VM on the Edge Gateway. vCC Node and vCC Server will also need to access internet therefore source NAT (SNAT) rule must be created for them. We could actually get away with just one external IP address: we could use port forwarding for the VAMI interface of each VM runnning on port 5480 (or we could even configure them over console from another VM with supported browsed deployed in the cloud). Please refer to my other post linked at the beginning of the article for the advanced networking information. In my lab I have luxury of 3 external IP addresses represented by 10.0.2.151-10.0.2.153 range.
   
9. Next we need to create firewall rules. As I already mentioned we need TCP port 5480 for VAMI interface. We also need TCP port 443 for vSphere Client connectivity to vCenter and TCP port 443 for incoming and outgoing traffic for vCloud Connector Node.
   
10. Now we can start the vApp and start configuring the VMs. I will skip the vCC Server and Node configuration and will focus on the vCenter Appliance part.
11. The inital configuration of vCenter Appliance is done via browser pointing to the 5480 port. In my case I am accessing the external NATed IP: https://10.0.2.151:5480. Default login is ‘root’ and password ‘vmware’.
12. After accepting EULA on the Configure Options screen I set custom configuration
    
13. The I chose embedded database and embedded SSO and did not enable Active Directory.
14. After the vCenter Server service (together with database and SSO) is started (which takes a while) do not forget to change the default password and optionally disable not needed services.
    
15. Now we can register vCC Server with this newly deployed vCenter Server. As they are on the same org VDC network I am using vCenter internal IP address (192.168.1.2).
    
16. That’s it. Now we can download vSphere Client and connect to the external IP address of vCenter Server and access the vCloud Connector Plugin in the Solutions and Applications section of the vCenter home page.
    

Licensing

There is one catch. Unlike the standard edition of vCloud Connector, vCenter Server is a licensed product. We can use the evaluation version for 60 days but what happens then? It turns out that even with expired license of vCenter Server you can still access the vCloud Connector plugin. So from technical standpoint it is possible to use vCenter Server without a license with vCloud Connector. Not sure what are the legal implications though (IANAL).

Real life example: 8 host cluster where each host has two 8 core @ 2.899 GHz CPUs and 384 GB RAM. Theoretically this should result in 371072 MHz CPU capacity and 3072 GB RAM however Resource Allocation tab of the cluster in vSphere client shows that only 330424 MHz and 2988 GB RAM is the total cluster capacity.

There is a KB article 1033443 describing the behavior with a title almost as long as the whole article: Cluster level memory capacity on Resource Allocation tab is less than the sum of the memory available for virtual machines for ESX hosts in the cluster, that unfortunately does not explain why and how much resources are missing.

As already hinted above, vmkernel processes are reserving some resources for themselves. If you select in vSphere Client a single host and go to Configuration > System Resource Allocation you will see value for System Resource Reservation – by default 301 MHz for CPU and 0 MB RAM.

However this view does not show the whole story – if you change from Simple to Advanced view (top right) you will be presented with a tree of resource pools each with their own reservations.

There is a host resource pool at the root of the tree which has all the theoretical physical resource available as reservation which also equals its limit. However then there are 4 children resource pools at the same level:

• idle, which is always empty
• system (this is the one we could edit on the previous page) containing low level kernel, driver and similar sub-resource pools for each process
• vim, which contains sub-resource pools for host management processes (hostd, vpxa, DCUI, …) which used to run in Console OS in the ESX 4 classic times.
• user, which is available for the VMs deployed on the host

All these 2nd level resource pools and all their children resource pools have expandable reservations, which means that if one of the children will request more resources that are available in the resource pool the resource pool will try to get more resources from its parent. And the top parent is the host resource pool. The system processes and management (VIM) processes are started immediately when the host boots up before VM workloads are placed on the host therefore take the part of the available host resources for themselves.

You can easily see that some processes like hostd or vpxa reserve relatively significant amount of resources. The relativity depends on the size of the host – in my small lab environment 36% of CPU and 20% of RAM resources were not available to be reserved for VMs. In big environments as was in the example above, the CPU overhead is about 11% but memory only 3%.

It should be also noted that with more and more intelligent hypervisor (VXLAN VTEP, vApp firewalling, antivirus inspection, vSAN, etc.) the overhead will go up and capacity planning should include it.

If you are familiar with the vSphere equivalent exam VCAP Datacenter Administration (VCAP-DCA) then you will find it very similar. It is 100% hands on exam with a series of tasks that have to be done on live vSphere/vCloud/Chargeback infrastructure. You start at a control desktop and then from there you get to the lab environment using any client of your choice (SSH, RDP, web browser, vSphere Client).

It is important to note that the lab environment is based on vCloud Director 5.1 (contrary to VCAP-CID where all the questions were based on vCloud Director 1.5). The questions follow the exam blueprint (as usually) quite well so I would recommend to review the blueprint before the exam. As I do vCloud for living it is hard for me to recommend training or studying resources – I thing very good hands on experience with vCloud Director, vShield Manager, Chargeback and RHEL is necessary to be successful. Only hard prerequisite is to be VCP and it does not matter if VCP-DCV (the former vSphere VCP-DV), VCP-DT (View) or VCP-Cloud. If you are not VCP-Cloud passing VCAP-CIA (or VCAP-CID) will give you VCP-Cloud certification by default.

The 29 questions are supposed to be solved in sequence as they are sometimes related and it is not possible to easily jump from one to another. You have to go forward or backward one question at a time. Some questions might be skipped though. Standard VMware documentation in PDF form is accessible during exam however VMware KB is not.

I had major problem with latency which showed its ugly head when typing “vmware1!” password which was universally used. The last character ‘!’ requires two keystroke pressed at the same time which did not work most of the time. That was pretty annoying during tasks that took a few minutes and failed because of wrong password. At the end I used copy – paste for the password – knowing this upfront it would save me quite a lot of time. Besides confusing wording of another question I had no real issues with the lab environment.

I have to say I really enjoyed the exam tasks and was impressed how the lab was set up. It is obvious it required some effort from the exam makers and I applaud this. I commented on my issues and as it was a beta it the final release might be different.

I am now eager for the result in order to take the next step to VCDX-Cloud.

In this post I will provide short step-by-step description of all the necessary steps that you as the vCloud Organization Administrator must take. The assumption is that you have on premise Horizon Workspace integrated with company Active Directory and want to use it for connecting private or public vCloud Director organizations.

1. Download Horizon Identity provider metadata XML file from: https://<horizon_workspace_URL>/SAAS/API/1.0/GET/metadata/idp.xml
2. In the target cloud go to Administration > Settings > Federation menu and check Use SAML Identity Provider and upload the idp.xml file
3. Still on the same page regenerate the certificate and click apply
4. Download the certificate from the url: https://<vcloud_URL>/cloud/org/<orgname>/saml/metadata/alias/vcd
5. Log out from the cloud
6. Log back in, you will need to change the URL to go directly to the local authentication: https://<vcloud_URL>/cloud/org/<orgname>/login.jsp
7. In the Administration > Members > Users (or Groups) import Users (or Groups) by clicking the icon with arrow. Change the Source to SAML and type the user names or group names.
8. Back in Horizon Workspace admin interface create a new Web Application in the catalog
9. Fill in the following data:
   • Authentication Profile: SAML 2.0 POST profile
   • Login Redirection URL: https://<vcloud_URL>/cloud/org/<orgname>/
   • Check: Include Destination
   • Check: Sign Response
   • Check: Sign the Assertion
   • Configure via Metadata XML
   • Paste the certificate from point 4 into the Meta-data XML box
   • Add Attribute Mapping as seen in the screenshot
     
     
   • Save the page
10. Edit the newly created Web Application and assign Entitlements (either specific users or a group). These should be the same users as in step 7.
11. Now log into the Horizon as the entitled user and click the application icon. You should now get direct access into the vCloud Director.

Edit 2 July 2013: In order to get SAML Groups working following is needed.

In step 9 create also group mapping. The group name must be hardcoded, but that should not be such a problem as a different web application in Horizon Workspace can be created for each group/role mapping. I have created hardcoded mapping to group name OrgAdministrators.

Then in step 7 the group can be imported and the correct role assigned.

Compatibility

vCloud Connector 2.0 (vCC) is backward compatible with vCloud Director 1.5, however some advanced feature like Content Sync will work in such deployments with limitations (modified templates are not removed, but added instead with timestamp in the name) or will not work at all (Stretch Deploy). vCloud Connector 1.5 does not work with vCloud Director 5.1.

Architecture and Network Flows

The architecture is similar to vCC 1.5, but the ports have changed (port 8443 not needed anymore). Here is the picture from the Installing and Configuring vCC 2.0 guide and let me dive deeper into some of the network flows (black circled numbers).

Flow 1

Although vCC Server has a web interface (VAMI) available at port 5480, the interface can be used only for appliance configuration and setting up connectivity to vCC Nodes. Internet Explorer is recommended to use here as Firefox or Chrome did not display some VAMI interface parts properly. The vCC Advanced Edition license key is entered here and SSL certificates related to the vCC Client – vCC Server communications can be enabled or generated and uploaded here as well. VAMI interface always uses https (on port 5480) with self signed certificate by default that cannot be replaced via the GUI, but if required it can be replaced in the following file on the appliance:

/opt/vmware/etc/lighttpd/server.pem

vCC end-users will use either vSphere client (the .NET version, as web client is not supported yet) or vcloud.vmware.com portal for actual management of the vApp transfers and other vCC features.

Although the picture shows port 80, if SSL is enabled, 443 will be used instead. If the replaced certificates do not use intermediate CA, the web interface cannot be used for their import and java keytool command must be used instead from appliance CLI as described in the installation guide. For some reason I was not able to create the private key with the GUI, but java keytool did the job.

Flow 2

vCC Server needs to be able to reach all vCC Nodes so the arrow should be also between vCC Server and the vCC Node in the destination cloud. Again the communication can go over port 80 or 443 depending on SSL configuration – this time on the vCC Nodes. The same caveat as with Server applies when replacing the certificates.

Note: Enabling SSL is recommended here as discussed below.

Flows 3 and 6/7

Prior attaching vCC Nodes to vCC Server, the Nodes need to be configured to connect to a cloud (vCenter or vCloud Director). SSL (port 443) is always used, but if you do not want to enable the Ignore SSL Cert checkbox, vCenter or vCloud need to have CA signed certificates. If you are using enterprise CA, you have to import the CA root certificate to a different keystore than the one available in the GUI as described in KB 2045007.

Flow 5

For the inter node communication, one node is designated as the Controller. The Controller initiates the connection. Which node is picked as the Controller depends on the Public checkbox setting in the vCC Node registration at vCC Server.

Public vCC Node

It is expected that the non Public vCC node is unreachable from outside and therefore has to be the initiator of the communication between nodes and is therefore the Controller. So the Controller Node works either in push or pull mode. If the other Node has SSL enabled (see Flow 2), https port 443 is used for the transfer between nodes. If the other Node does not have SSL enabled the transfer will fail.

Shared Node

This is a new feature for vCloud Director deployments when the provider can deploy shared (multitenant) node and connect it to provider’s cloud. The tenants then do not have to deploy their own nodes inside their organization VDCs with all the troubles of securing connectivity and appropriate transfer storage.

In highly secure public clouds it gets tricky where and how to deploy the shared node. It cannot be load balanced, but provider can deploy multiple shared nodes and give their IP addresses only to specific groups of tenants. The node should be as close as possible to the vCloud API somewhere in DMZ accessible from the internet but if Web Application Firewall is used it should still be in-between the node and API as it could be used as an attack vector to other organizations. The node does not keep any vCloud credentials for communication with vCloud API. Those are transferred by vCC Server so again SSL should be enabled (see Flow 2).

Content Sync

This is a new feature, great for maintenance of catalogs in various clouds or might be also used by the provider for management of public catalog directly from vSphere where a vSphere folder is automatically synced with a vCloud Director catalog. Note however that vCC Advanced Edition license is needed which currently cannot be obtain with provider VSPP license, but only through vCloud Suite bundles.

The default polling interval for synchronization is 6 hours. It can be changed but the change is unsupported. Following file can be edited on the vCC Server:

/usr/local/tcserver/vfabric-tc-server-standard/server/webapps/agent/WEB-INF/spring/appServlet/task.xml

Look for <property name=”jobExecutionIntervalInMinutes” value=”360″ />.

Although the documentation states that ports 8443 and 8080 are used for Content Sync, my understanding is that they are used only internally on the vCC Server.

Stretch Deploy

This is again a new but licensed feature which enables migration of VMs between clouds without needing to change their IPs or MAC addresses thanks to a VPN connection (not VXLAN as often confused) which is established between the original and destination cloud. The SSL VPN is established between Edges on the vApp networks so this is quite different from the Site-to-Site IPSec VPN between Edge Gateways even though it shows up in vShield Manager in the IPSec VPN section. Its configuration is not exposed in the vCloud Director GUI at all.

Stretch Deploy Site-to-Site SSL VPN

There is quite a long list of prerequisites to get this working and some of them are out of tenant’s controls as they relate to the provider’s vCloud Director architecture – e.g. vSphere and vCloud Network and Security versions and type of distributed virtual switch used. Stretch Deploy will not work with Cisco Nexus 1000V switch. The tenant most likely will not know which switch the provider is using until he will experience following error:

Unable to update network “Stretched_VM_network”.
java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (100): A specified parameter was not correct.
selectionSet.dvsUuid

The source and transferred destination VMs need to be connected to vSphere Distributed Switch 5.1 as the unclaimed traffic needs to be sent over the SSL tunnel and this is not currently supported with other virtual switches.

The Stretch Deploy process is relatively complicated with many actions that are happening in the background. This is all abstracted from the user as he can start the process easily from vCC GUI. However if you want to end the stretch deploy by removing the remote VM, or bring it home back this must be done manually.

Stretch Deploy activities as seen by the vCenter (note both source and destination clouds were managed by the same vCenter)

Delete Stretch Deployed VM

• stop the remote vApp, this will terminate the VPN connection and destroy the vApp Edge
• delete the remote vApp
• delete the IPSec configuration in the vSphere cloud Edge in vShield Manager
• delete vCenter custom attributes of the VM which was stretched deployed (DatacenterExtendedEntityId, DatacenterExtensionRole)

Bring Home the Stretch Deployed VM

This must be done by running a script from the vCC Node managing the private cloud. So an access to the Node is needed, the script needs to be untared and quite a lot of information must be typed in when executed and their correctness is not verified until they are all typed in. This is definitely not task for an average user and I expect this part might be improved in later releases.

When vCenter is connected to vCloud Director a VC Proxy service is started on one of the cells. The service is responsible for monitoring of active vCenter tasks and inventory updates which are then shared with other cells. Unless there is a network partition between the cells there is always one vCenter proxy service for one vCenter. Multiple VC Proxies can run on one cell. You can see which cell is running the VC Proxy service at the vCenter screen in the vCloud Director Admin interface.

The screenshot shows two vCenters connected to vCloud Director with one having its vCenter proxy on vcloud1 cell and the second on vcloud2 cell.

If the VC Proxy service is not running most of the activities in the vCloud Director that require vCenter will not work properly. For example simple creation of a vApp with one VM will fail with message:

Folder vApp_system_34 (8ce90b57-da8b-4714-914b-5073457155b0) does not exist in our inventory, but vCenter Server claims that it does.

This is because the inventory listener on the VC Proxy was not running and vCloud Director could not verify successful creation of vApp folder in vCenter. When a cell with VC Proxy service dies the service fails over to a surviving cell. However that failover takes 5 minutes which is govern by vcloud:vcloud.heartbeat.failoverTimeoutMsecs property (stored in vCloud Director database). I am not aware if it is supported to change this value.

Anyway in order to shutdown a cell gracefully we need to move the VC Proxy service to another cell. This can be done by simple reconnect of vCenter and the move is very quick without any disruption of the running tasks.

Reconnect vCenter

I have observed that if possible different cell then the original and the least loaded (in terms of number of VC Proxy services) is chosen. This is also good for manually distributing the load if there are multiple vCenters and multiple cells (good practice is to have at least N+1 cells, where N is number of vCenters).

So what should be the correct graceful cell shutdown procedure?

1. Make sure the cell is not running any VC Proxy service. No checkmark should be in the vCenter column of the Cloud Cells inventory in the vCloud Director Admin interface.If yes, then reconnect vCenters that have VC Proxy running on the cell.
2. Quiesce the cell with the cell-management-tool:

   $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –quiesce true

   where <user> is vCloud administrator username

3. Monitor the number of outstanding active tasks on the cell and wait until it reaches 0.

   $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –status
   Job count = 0
   Is Active = false

4. Shutdown the cell. This can be done also with cell-management-tool. What I noticed is that it takes multiple attempts (usually two), as the first time only the watchdog service is terminated.

   # $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –shutdown

   # service vmware-vcd status

   vmware-vcd-watchdog is not running
   vmware-vcd-cell is running

   # $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –shutdown
   # service vmware-vcd status

   vmware-vcd-watchdog is not running
   vmware-vcd-cell is not running