vCloud Connector and Offline Data Transfer

Offline Data Transfer (ODT) is a feature of vCloud Connector that allows migration of VMs from customer own datacenter to vCloud Air with NAS appliance which is shipped via regular mail. The point is to avoid slow wide area network connectivity and leverage awesome bandwidth but slow latency of sneakernet.

Have you ever wondered why it is supported only with vCloud Air and not with any public or private cloud based on vCloud Director? Well I am going to lay down the whole process here in this blog post so nothing is stopping anyone testing this feature on your own.

Let me first paste picture from the manual which describes in high level how the process works:

Offline Data Transfer ProcessvCloud Connector (vCC) is leveraged to manage the whole process. The customer (on left) deploys his own vCloud Connector Server and Node which he attaches to his on premises infrastructure (vSphere based). He then requests the ODT service. The provider will deploy ODT node in the public cloud (on right) and also its own vCC Server to manage it. Regular NAS appliance is prepared – its only purpose is to provide storage capacity which is fast and reliable enough via NFS protocol and can be easily packaged and shipped.

Customer mounts it to his vCC Node (to a directory via NFS mount). Both the ODT and vCC Nodes are registered in his vCC Server. Then via the traditional vSphere Client and vCC Plugin only the local vSphere environment (here it differs from the traditional vCC transfer).

vSphere in vCC

The actual export is done by selecting the objects to export (templates, vApps or VMs) and clicking the small Offline Data Transfer icon: ODT IconMount path is entered and links and credentials to the target Cloud and ODT node. There is also option to select if a particular VM should be deployed and connected to a network. These steps above are all described in the manual here.

But what about the provider side of the whole process?

ODT Node

ODT Node is actually regular vCloud Connector Node tweaked by running configureSneakernetNode.sh script which can be found in /opt/vmware/hcagent/scripts folder on the Node VM itself.

The ODT needs to have network access to the vCenter Server (and ESXi hosts) of the target vCloud VDC environment.

Import

The actual import is done via provider vCloud Connector server which is again the regular vCloud Connector server with no tweaks this time. The ODT Node is registered there which enables import menu in the vCC plugin GUi in vSphere Client. The shipped NAS appliance must be mounted to the ODT Node and the ODT URL and mount path is entered in the Import Wizzard. The actual physical connection of the NAS appliance can be done using dedicated VLAN with point to point connection of the second ODT network interface.

ImportNext we need to pick the target vCenter Server and a credentials for it. ODT Node will import offline VMs which are stored as encrypted OVFs on the NAS appliance into the target vCenter Server. To do that it needs a big enough datastore and a dummy network in order to connect the imported VMs temporarily to it. Once that is done VMs are imported by vCloud Director to the target VDCs, catalogs and networks. The provider needs to have big enough datastore and create dummy standard switch port group on every host with name ‘VM Network’. This network does not need to have external access.

As you can see contrary to the regular internet vCloud Connector transfer where the VM is transfer from the original environment via on-prem node to public node to vCloud Director (through its API and transfer storage – see here for more detail) the transfer does not go through the vCloud Director cells and its transfer storage at all. This is possible thanks to handling the final step of the process by the provider himself (he has vCenter Server access) and makes also the transfer faster (potentially one less step). On the other hand this brings some security and operational process challenges (physical access to management network, vCenter credentials) which must be properly addressed.

vCloud Connector in Multisite Environment with Single VC

I have received an interesting question. Customer has single vCenter Server environment with large number of sites. Can they use vCloud Connector Content Sync to keep templates synced among the sites?

The answer is yes but the set up is not so straight forward. vCloud Connector client does not allow registration of multiple nodes pointing to the same end-point (vCenter Server). The workaround is to fool vCloud Connector by using different FQDNs for the same end-point (IP address). Example: vcenter01.fojta.com, vcenter01a.fojta.com, vcenter01b.fojta.com all pointing to the same IP address. This will obviously impact SSL certificate verification which must be turned of or the vCenter Server certificates must include all those alternative names in S.A.N. attribute.

So the customer would deploy vCloud Connector node in each site, register it with the central vCenter Server always using different FQDN. Then in vCloud Connector client each site would look like a different vCenter Server and the Content Sync could be set up.

vCloud Connector 2.0 and SSL Certificate Replacement

In my previous post about vCloud Connector I went through the various data flows and also explained that it is necessary to have SSL enabled both on the vCC Server and vCC nodes. It is possible to use self-signed certificates or also it is possible to upload via vCC Server/Node VAMI GUI (Virtual Appliance Management Interface) publicly trusted and signed certificates. However when you want to use certificates provided by your own Enterprise Certificate Authority the process is not so straight forward and requires using command line.

The problem statement is following: I have my own Enterprise Certificate Authority which is not publicly trusted and I have issued certificates for my vCC Server and all Nodes without any intermediate certificate. How do I import them?

The vCC VAMI GUI supoprts only import of certificate chain that consist of root>intermediate>certificate. As I do not have intermediate certificate I have to use JAVA keytool command. There are two keystores: cacerts for trusted root certificates and tcserver.jks for the other certificates. My Enterprise CA root certificate must be imported into the former one and the vCC Server/Node certificates to the latter.

Here is the exact procedure:

vCloud Connector Server

1. Log in into the vCC Server console as root and delete the self signed hcserver certificate from tcserver.jks keystore. Note the keystore password (changeme) is hardcoded – do not try to change it.

/usr/java/latest/bin/keytool -delete -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

2. Create new hcserver certificate, where CN (first and last name) is the vCC Server FQDN. Use the same password for the certificate as the keystore password (just hit RETURN when asked).

/usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

What is your first and last name?
 [Unknown]:  vccserver.fojta.com
What is the name of your organizational unit?
  [Unknown]:vCloud Connector Server
What is the name of your organization?
  [Unknown]:  fojta.com
What is the name of your City or Locality?
  [Unknown]:  Prague
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:  CZ
Is CN=vccserver.fojta.com, OU= vCloud Connector Server, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
  [no]:  yes
 Enter key password for <hcserver>
        (RETURN if same as keystore password):

3. Now we have to create the Certificate Signing Request. We can do it either from command line or from the VAMI GUI (Server>SSL>Generate and download CSR)

/usr/java/latest/bin/keytool -certreq -alias hcserver -file hcserver.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

4. Sign the certificate signing request (hcserver.csr) with your Enterprise CA. I have used the Subordinate Certification Authority certificate template. Upload the signed certificate to the vCC server as file hcserver.cer.

5. Before importing hcserver.cer certificate we need to import the CA root certificate otherwise it would not be able to build trusted chain. Obtain the CA root certificate (in my case named fojta-dc-CA.cer and import to cacerts keystore with following command). The alias must be unique. Note the certificate password (changeit) is hardcoded, do not try to change it (no pun intended).

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Now we can import the hcserver certificate

/usr/java/latest/bin/keytool -import -alias hcserver -file hcserver.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

7. Verify that the import was successful.

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcserver, May 10, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): FC:A3:29:96:0D:CD:E2:04:3D:0F:E9:B6:8B:A0:6F:B8:C8:BF:3E:61

vCloud Connector Node

The process is identical to the vCloud Connector Server certificate replacement with just minor changes. The certificate alias is not hcserver but hcagent and the tcserver.jks keystore location is different. Therefore just briefly:

1. Delete selfsigned hcagent certificate
/usr/java/latest/bin/keytool -delete -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

2. Create new hcagent certificate

/home/admin # /usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

What is your first and last name?
[Unknown]:  vccnode.fojta.com
What is the name of your organizational unit?
[Unknown]:vCloud Connector Node
What is the name of your organization?
[Unknown]:  fojta.com
What is the name of your City or Locality?
[Unknown]:  Prague
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]:  CZ
Is CN=vccnode.fojta.com, OU=vCloud Connector Node, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
[no]:  yes

Enter key password for <hcagent>
(RETURN if same as keystore password):

3. Create CSR or download from VAMI GUI: Node>SSL>Generate and download CSR

/usr/java/latest/bin/keytool -certreq -alias hcagent -file hcagent.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

4. Sign hcagent.csr with your CA => hcagent.cer

5. Import CA root certificate (fojta-dc-CA.cer)

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Import hcagent certificate

/usr/java/latest/bin/keytool -import -alias hcagent -file hcagent.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

7. Verify

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcagent, May 10, 2013, PrivateKeyEntry
Certificate fingerprint (SHA1): 9E:63:B1:BC:91:FE:7D:84:FC:8C:66:24:B9:1B:B9:73:80:5D:AC:87

Enabling SSL

Now we should be able both on vCC Server and all vCC Nodes to enable SSL (in Server/Node tab, SSL subtab click Enable SSL button). When returning the the SSL tab it seems vCC is not able to get the current SSL status correctly. vCC Server displays an error and vCC Node nothing – not really sure why is this happening, just ignore it.

Enable SSL

Now we should be able in the Nodes tab on the vCC Server edit the Node configuration and uncheck Ignore SSL Certificate. If the certificate replacement was successful, the Node Status shoud be Up.

Ignore SSL Certificate

For more details rather refer to the online vCloud Connector documentation as it contains more information than the PDF docs.

vCloud Connector: Public Cloud Transfers with no Private vSphere Environment

I have already blogged quite extensively about vCloud Connector – a tool for transfers of VMs, vApps and catalogs between public and private vSphere or vCloud Director based clouds. This post is dedicated to one particular use case, let’s call it ‘Developer Use Case‘. Here the user is not VI Admin, but a developer that wants to deploy his apps to public clouds. He has no access to private vSphere environment and obviously to vCenter Server.

vCloud Connector (vCC) consists of vCloud Connector Server, vCloud Connector Nodes and vCloud Connector Client. In the past the vCloud Connector was accessible either through vCenter Server plugin or through web interface at http://vcloud.vmware.com/connector. However as of April 2nd 2013 VMware discontinued the web interface access which means that vCenter Server plugin is currently the only option to use vCloud Connector.

The web portal was great for the Developer Use Case. The developer did not need to have access to vSphere, vCenter, vSphere client or even a Windows machine and could use vCloud Connector from his Mac or Linux browser.

What now follows in this article is basically a description how the Developer Use Case can still be fulfilled even after April 2nd. The idea is to use vCenter Server Appliance and deploy it to the cloud together with vCloud Connector Server. This vCenter Server Appliance will not manage any ESX hosts and will be basically used only for the instantiation of the vCloud Connector Client interface. The thick (.NET) vSphere Client is still needed (as at the moment there is no vCC plugin for vSphere Web Client), this also means a Windows OS, so the developer will need either physical Windows desktop, or a virtual one on his PC or running also in the cloud and accessible via RDP.

How to deploy vCenter Server to the public vCloud Director cloud

  1. From VMware website download vCenter Server Appliance. I have used the version 5.1.0b which comes as one large OVA file.
  2. As we cannot import OVA file to vCloud Director we first need to unzip the file to get the OVF format. This can be done easily by adding .zip extension to the downloaded file and using WinZip or similar utility.
  3. Import the OVF file into your organization catalog.
  4. As I also had vCC Server and vCC Node in the catalog I deployed them together into one vApp.
    vCloud Connector vApp
  5. After accepting EULA, selecting Storage Profile and setting hostnames it is important to put the VMs on one internet routable network and manually assign IP addresses from the static pool of the organization VDC network. We cannot just rely on Guest Customization as the IP address assignment is part of the vApp property which is applied when the vApp is deployed. So in my case I used (the default) 192.168.1.0/24 subnet for the org VDC network, where IP 192.168.1.1 was used for the internal interface of the Edge Gateway and IPs 192.168.1.2-192.168.1.4 were used for vCenter Server, vCloud Connector Server and vCloud Connector Node.IP Assignment Static - Manual
  6. On the next page we are presented with the vApp properties. Here we have to again manually assign the correct IP addresses as specified in the previous step. As I am deploying 3 imported vApps at once it looks quite confusing as the properties of each are merged into one screen. The default gateway address is the Edge Gateway internal interface and subnet is 255.255.255.0.Imported vApp Network Properties
  7. After the deployment is finished we will get vCloud Connector vApp which looks like this:vCloud Connector vApp Networking Diagram
  8. We will need to access all three VMs from the internet to configure their Virtual Appliance Management Interface (VAMI) which runs on TCP port 5480. If you have 3 external IP addresses you can set up destination NAT (DNAT) rules for each VM on the Edge Gateway. vCC Node and vCC Server will also need to access internet therefore source NAT (SNAT) rule must be created for them. We could actually get away with just one external IP address: we could use port forwarding for the VAMI interface of each VM runnning on port 5480 (or we could even configure them over console from another VM with supported browsed deployed in the cloud). Please refer to my other post linked at the beginning of the article for the advanced networking information. In my lab I have luxury of 3 external IP addresses represented by 10.0.2.151-10.0.2.153 range.
    NAT rules
  9. Next we need to create firewall rules. As I already mentioned we need TCP port 5480 for VAMI interface. We also need TCP port 443 for vSphere Client connectivity to vCenter and TCP port 443 for incoming and outgoing traffic for vCloud Connector Node.
    Firewall Rules
  10. Now we can start the vApp and start configuring the VMs. I will skip the vCC Server and Node configuration and will focus on the vCenter Appliance part.
  11. The inital configuration of vCenter Appliance is done via browser pointing to the 5480 port. In my case I am accessing the external NATed IP: https://10.0.2.151:5480. Default login is ‘root’ and password ‘vmware’.
  12. After accepting EULA on the Configure Options screen I set custom configuration
    vCenter Configure Options
  13. The I chose embedded database and embedded SSO and did not enable Active Directory.
  14. After the vCenter Server service (together with database and SSO) is started (which takes a while) do not forget to change the default password and optionally disable not needed services.
    vCenter Services
  15. Now we can register vCC Server with this newly deployed vCenter Server. As they are on the same org VDC network I am using vCenter internal IP address (192.168.1.2).
    vCC Server vCenter Server registration
  16. That’s it. Now we can download vSphere Client and connect to the external IP address of vCenter Server and access the vCloud Connector Plugin in the Solutions and Applications section of the vCenter home page.
    vCC Plugin

Licensing

There is one catch. Unlike the standard edition of vCloud Connector, vCenter Server is a licensed product. We can use the evaluation version for 60 days but what happens then? It turns out that even with expired license of vCenter Server you can still access the vCloud Connector plugin. So from technical standpoint it is possible to use vCenter Server without a license with vCloud Connector. Not sure what are the legal implications though (IANAL).

Expired vCenter Server License

vCloud Connector 2.0 Observations

I have been playing with vCloud Connector 2.0 which is already a third release of the tool that enables VM (vApp) transfers between various VMware clouds based on vSphere or vCloud Director. Here follows a bunch of notes that I came up with that might help others. Note I expect that the reader is familiar with basic functionality and architecture of vCC.

Compatibility

vCloud Connector 2.0 (vCC) is backward compatible with vCloud Director 1.5, however some advanced feature like Content Sync will work in such deployments with limitations (modified templates are not removed, but added instead with timestamp in the name) or will not work at all (Stretch Deploy). vCloud Connector 1.5 does not work with vCloud Director 5.1.

Architecture and Network Flows

The architecture is similar to vCC 1.5, but the ports have changed (port 8443 not needed anymore). Here is the picture from the Installing and Configuring vCC 2.0 guide and let me dive deeper into some of the network flows (black circled numbers).

vCC Data Flows

Flow 1

Although vCC Server has a web interface (VAMI) available at port 5480, the interface can be used only for appliance configuration and setting up connectivity to vCC Nodes. Internet Explorer is recommended to use here as Firefox or Chrome did not display some VAMI interface parts properly. The vCC Advanced Edition license key is entered here and SSL certificates related to the vCC Client – vCC Server communications can be enabled or generated and uploaded here as well. VAMI interface always uses https (on port 5480) with self signed certificate by default that cannot be replaced via the GUI, but if required it can be replaced in the following file on the appliance:

/opt/vmware/etc/lighttpd/server.pem

vCC end-users will use either vSphere client (the .NET version, as web client is not supported yet) or vcloud.vmware.com portal for actual management of the vApp transfers and other vCC features.

Although the picture shows port 80, if SSL is enabled, 443 will be used instead. If the replaced certificates do not use intermediate CA, the web interface cannot be used for their import and java keytool command must be used instead from appliance CLI as described in the installation guide. For some reason I was not able to create the private key with the GUI, but java keytool did the job.

Flow 2

vCC Server needs to be able to reach all vCC Nodes so the arrow should be also between vCC Server and the vCC Node in the destination cloud. Again the communication can go over port 80 or 443 depending on SSL configuration – this time on the vCC Nodes. The same caveat as with Server applies when replacing the certificates.

Note: Enabling SSL is recommended here as discussed below.

Flows 3 and 6/7

Prior attaching vCC Nodes to vCC Server, the Nodes need to be configured to connect to a cloud (vCenter or vCloud Director). SSL (port 443) is always used, but if you do not want to enable the Ignore SSL Cert checkbox, vCenter or vCloud need to have CA signed certificates. If you are using enterprise CA, you have to import the CA root certificate to a different keystore than the one available in the GUI as described in KB 2045007.

Flow 5

For the inter node communication, one node is designated as the Controller. The Controller initiates the connection. Which node is picked as the Controller depends on the Public checkbox setting in the vCC Node registration at vCC Server.

Public vCC Node
Public vCC Node

It is expected that the non Public vCC node is unreachable from outside and therefore has to be the initiator of the communication between nodes and is therefore the Controller. So the Controller Node works either in push or pull mode. If the other Node has SSL enabled (see Flow 2), https port 443 is used for the transfer between nodes. If the other Node does not have SSL enabled the transfer will fail.

Shared Node

This is a new feature for vCloud Director deployments when the provider can deploy shared (multitenant) node and connect it to provider’s cloud. The tenants then do not have to deploy their own nodes inside their organization VDCs with all the troubles of securing connectivity and appropriate transfer storage.

In highly secure public clouds it gets tricky where and how to deploy the shared node. It cannot be load balanced, but provider can deploy multiple shared nodes and give their IP addresses only to specific groups of tenants. The node should be as close as possible to the vCloud API somewhere in DMZ accessible from the internet but if Web Application Firewall is used it should still be in-between the node and API as it could be used as an attack vector to other organizations. The node does not keep any vCloud credentials for communication with vCloud API. Those are transferred by vCC Server so again SSL should be enabled (see Flow 2).

Content Sync

This is a new feature, great for maintenance of catalogs in various clouds or might be also used by the provider for management of public catalog directly from vSphere where a vSphere folder is automatically synced with a vCloud Director catalog. Note however that vCC Advanced Edition license is needed which currently cannot be obtain with provider VSPP license, but only through vCloud Suite bundles.

The default polling interval for synchronization is 6 hours. It can be changed but the change is unsupported. Following file can be edited on the vCC Server:

/usr/local/tcserver/vfabric-tc-server-standard/server/webapps/agent/WEB-INF/spring/appServlet/task.xml

Look for <property name=”jobExecutionIntervalInMinutes” value=”360″ />.

Although the documentation states that ports 8443 and 8080 are used for Content Sync, my understanding is that they are used only internally on the vCC Server.

Stretch Deploy

This is again a new but licensed feature which enables migration of VMs between clouds without needing to change their IPs or MAC addresses thanks to a VPN connection (not VXLAN as often confused) which is established between the original and destination cloud. The SSL VPN is established between Edges on the vApp networks so this is quite different from the Site-to-Site IPSec VPN between Edge Gateways even though it shows up in vShield Manager in the IPSec VPN section. Its configuration is not exposed in the vCloud Director GUI at all.

Stretch Deploy Site-to-Site SSL VPN
Stretch Deploy Site-to-Site SSL VPN

There is quite a long list of prerequisites to get this working and some of them are out of tenant’s controls as they relate to the provider’s vCloud Director architecture – e.g. vSphere and vCloud Network and Security versions and type of distributed virtual switch used. Stretch Deploy will not work with Cisco Nexus 1000V switch. The tenant most likely will not know which switch the provider is using until he will experience following error:

Unable to update network “Stretched_VM_network”.
java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (100): A specified parameter was not correct.
selectionSet.dvsUuid

The source and transferred destination VMs need to be connected to vSphere Distributed Switch 5.1 as the unclaimed traffic needs to be sent over the SSL tunnel and this is not currently supported with other virtual switches.

The Stretch Deploy process is relatively complicated with many actions that are happening in the background. This is all abstracted from the user as he can start the process easily from vCC GUI. However if you want to end the stretch deploy by removing the remote VM, or bring it home back this must be done manually.

Stretch Deploy activities as seen by the vCenter (note both source and destination clouds were managed by the same vCenter)
Stretch Deploy activities as seen by the vCenter (note both source and destination clouds were managed by the same vCenter)

Delete Stretch Deployed VM

  • stop the remote vApp, this will terminate the VPN connection and destroy the vApp Edge
  • delete the remote vApp
  • delete the IPSec configuration in the vSphere cloud Edge in vShield Manager
  • delete vCenter custom attributes of the VM which was stretched deployed (DatacenterExtendedEntityId, DatacenterExtensionRole)

Bring Home the Stretch Deployed VM

This must be done by running a script from the vCC Node managing the private cloud. So an access to the Node is needed, the script needs to be untared and quite a lot of information must be typed in when executed and their correctness is not verified until they are all typed in. This is definitely not task for an average user and I expect this part might be improved in later releases.