vCloud Connector 2.0 and SSL Certificate Replacement

In my previous post about vCloud Connector I went through the various data flows and also explained that it is necessary to have SSL enabled both on the vCC Server and vCC nodes. It is possible to use self-signed certificates or also it is possible to upload via vCC Server/Node VAMI GUI (Virtual Appliance Management Interface) publicly trusted and signed certificates. However when you want to use certificates provided by your own Enterprise Certificate Authority the process is not so straight forward and requires using command line.

The problem statement is following: I have my own Enterprise Certificate Authority which is not publicly trusted and I have issued certificates for my vCC Server and all Nodes without any intermediate certificate. How do I import them?

The vCC VAMI GUI supoprts only import of certificate chain that consist of root>intermediate>certificate. As I do not have intermediate certificate I have to use JAVA keytool command. There are two keystores: cacerts for trusted root certificates and tcserver.jks for the other certificates. My Enterprise CA root certificate must be imported into the former one and the vCC Server/Node certificates to the latter.

Here is the exact procedure:

vCloud Connector Server

1. Log in into the vCC Server console as root and delete the self signed hcserver certificate from tcserver.jks keystore. Note the keystore password (changeme) is hardcoded – do not try to change it.

/usr/java/latest/bin/keytool -delete -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

2. Create new hcserver certificate, where CN (first and last name) is the vCC Server FQDN. Use the same password for the certificate as the keystore password (just hit RETURN when asked).

/usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcserver -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

What is your first and last name?
 [Unknown]:  vccserver.fojta.com
What is the name of your organizational unit?
  [Unknown]:vCloud Connector Server
What is the name of your organization?
  [Unknown]:  fojta.com
What is the name of your City or Locality?
  [Unknown]:  Prague
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:  CZ
Is CN=vccserver.fojta.com, OU= vCloud Connector Server, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
  [no]:  yes
 Enter key password for <hcserver>
        (RETURN if same as keystore password):

3. Now we have to create the Certificate Signing Request. We can do it either from command line or from the VAMI GUI (Server>SSL>Generate and download CSR)

/usr/java/latest/bin/keytool -certreq -alias hcserver -file hcserver.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

4. Sign the certificate signing request (hcserver.csr) with your Enterprise CA. I have used the Subordinate Certification Authority certificate template. Upload the signed certificate to the vCC server as file hcserver.cer.

5. Before importing hcserver.cer certificate we need to import the CA root certificate otherwise it would not be able to build trusted chain. Obtain the CA root certificate (in my case named fojta-dc-CA.cer and import to cacerts keystore with following command). The alias must be unique. Note the certificate password (changeit) is hardcoded, do not try to change it (no pun intended).

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Now we can import the hcserver certificate

/usr/java/latest/bin/keytool -import -alias hcserver -file hcserver.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

7. Verify that the import was successful.

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/server/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcserver, May 10, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): FC:A3:29:96:0D:CD:E2:04:3D:0F:E9:B6:8B:A0:6F:B8:C8:BF:3E:61

vCloud Connector Node

The process is identical to the vCloud Connector Server certificate replacement with just minor changes. The certificate alias is not hcserver but hcagent and the tcserver.jks keystore location is different. Therefore just briefly:

1. Delete selfsigned hcagent certificate
/usr/java/latest/bin/keytool -delete -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

2. Create new hcagent certificate

/home/admin # /usr/java/latest/bin/keytool -genkey -validity 3650 -keyalg RSA -keysize 2048 -alias hcagent -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

What is your first and last name?
[Unknown]:  vccnode.fojta.com
What is the name of your organizational unit?
[Unknown]:vCloud Connector Node
What is the name of your organization?
[Unknown]:  fojta.com
What is the name of your City or Locality?
[Unknown]:  Prague
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]:  CZ
Is CN=vccnode.fojta.com, OU=vCloud Connector Node, O=fojta.com, L=Prague, ST=Unknown, C=CZ correct?
[no]:  yes

Enter key password for <hcagent>
(RETURN if same as keystore password):

3. Create CSR or download from VAMI GUI: Node>SSL>Generate and download CSR

/usr/java/latest/bin/keytool -certreq -alias hcagent -file hcagent.csr -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

4. Sign hcagent.csr with your CA => hcagent.cer

5. Import CA root certificate (fojta-dc-CA.cer)

/usr/java/latest/bin/keytool -import -alias fojta-dc-CA -file fojta-dc-CA.cer -keystore /usr/java/default/lib/security/cacerts -storepass changeit

6. Import hcagent certificate

/usr/java/latest/bin/keytool -import -alias hcagent -file hcagent.cer -trustcacerts -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

7. Verify

/usr/java/latest/bin/keytool -list -keystore /usr/local/tcserver/vfabric-tc-server-standard/agent/conf/tcserver.jks -storepass changeme

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

hcagent, May 10, 2013, PrivateKeyEntry
Certificate fingerprint (SHA1): 9E:63:B1:BC:91:FE:7D:84:FC:8C:66:24:B9:1B:B9:73:80:5D:AC:87

Enabling SSL

Now we should be able both on vCC Server and all vCC Nodes to enable SSL (in Server/Node tab, SSL subtab click Enable SSL button). When returning the the SSL tab it seems vCC is not able to get the current SSL status correctly. vCC Server displays an error and vCC Node nothing – not really sure why is this happening, just ignore it.

Now we should be able in the Nodes tab on the vCC Server edit the Node configuration and uncheck Ignore SSL Certificate. If the certificate replacement was successful, the Node Status shoud be Up.

For more details rather refer to the online vCloud Connector documentation as it contains more information than the PDF docs.

VMware View 5.1 and SSL Certificate Replacement

The procedure to replace SSL certificates has changed in recently released VMware View 5.1 and differs from 5.0 or earlier versions. The main difference is that native Windows certificate store is used. Also it is now necessary to replace or at least to verify self signed certificates otherwise the View infrastructure will not work properly.
Which servers need to replace the certificates? View Connection Managers, Security servers and View Composer. Also vCenter certificate must be replaced or validated.
Although the certificate replacement procedure is described in the manual, the description is very brief and it took me some time to figure it out. I also found an existing blog post which takes different angle here: http://my-virt.alfadir.net/2012/05/generate-view-5-1-certificat/.

The Setup

My lab configuration is following. View Composer is installed together with vCenter. I have two View Connection Managers; one for internal connections and one for external internet connections. I do not use Security server as I use port forwarding to the external View Connection Manager. All servers are in the domain with Enterprise CA which uses self signed certificate.

View Composer Certificate

My View Composer server is coexisting with vCenter so I did not need to generate new certificate. I just imported the vCenter certificate from C:\ProgramData\VMware\VMware VirtualCenter\SSL into the local Windows certificate (Personal) store via MMC Certificates Snap-in.

Select the .pfx file which contains both private key and the certificate.

Now we have to stop the View Composer process and run SviConfig command to replace the certificate:

C:\Program Files (x86)\VMware\VMware View Composer>SviConfig.exe -operation=replacecertificate -delete=false
and select the new certificate.

Start the View Composer process again and check the status in the View Administrator.

View Connection Servers

Here we have to generate the certificates. To do this I am again using the Certificates Snap-in. However prior to that I needed to give both of my Connection Server access to Web Server certificate template.
On my CA open the Certificate Templates Management Console Snap-in and open properties of Web Server certificate template.

Open the security tab and add both View Connection computers and give them read, write and enroll permissions.

Now we can go to each Connection Server Certificates Snap-in and right click, select All Tasks and Request New Certificate.

Select Active Directory Enrollment Policy and Web Server certificate template.

Now we have to add FQDN and additional info to the certificate. Click the More information is required … link.
Type in the Common name (FQDN), Country, Locality, Organization and any other info that will be visible inside the certificate. If your Connection server uses different internal (viewcs2.fojta.com) and public (public.url.com) Fully Qualified Domain Names add both and do the same for the Type: DNS field. The end result should look like this:

And do not forget to make private key exportable in the Private Key tab / Key options.

Click enroll and finish.
Now we should see the newly created certificate. Last thing we need to do is to change the certificate Friendly Name to vdm. This can be done in the certificate properties. Also we have to rename the original certificate (vdm.old)

Once this is done we can restart the View services.
Repeat for all other View Connection servers and check the result in the View Administrator.

View Clients

As I said at the beginning my CA uses self signed certificate so I have to make sure all the non-domain PCs I use to connect to my View desktops imported the CA Root certificate into the Trusted Root Certification Authorities store.

vCloud Director and SSL Certificates

There is always a time during production deployments of vCloud Director when the question of security comes up. One of the most basic things, how to harden vCloud Director is to correctly configure SSL certificates.

VCD Cell SSL Certificates

The communication between the end-user and vCloud Director cell (either GUI or API) is encrypted and by default self-signed certificates are used. The certificate replacement procedure is explained in the documentation in a few simple steps. The problem I have encountered during big vCloud deployments is that enterprise security teams have specific procedures how to create and distribute certificates which is different from those described in the documentation.

The default procedure is following:

• create untrusted certificates (private and public key) with JAVA keytool command (this must be done for vCloud Director GUI and console proxy)
• create certificate signing requests
• send the certificate signing requests to your Certification Authority
• import the Certificate Authority root certificate
• import signed certificates

In my case the certificates were created for me by the security team and I have received the private key in a .key file. On top of that the Certification Authority which signed the certificates was intermediate and was signed by two others. The chain was following: public Root CA -> intermediate CA1 -> intermediate CA2 -> VCD certificate.

vCloud Director JAVA keytool command does not allow private key import. Also the whole trusted chain for the certificate must be built so all the intermediate certificates are presented to the client browsers and the vCloud Director certificate can be validated. This has been achieved with the following procedure:

1. Concatenate all CA certificates to create the whole chain:

   
cat CA2.cer CA1.cer RootCA.cer > chain.crt

2. With openssl create PKCS12 keystore with the private key, certificate chain and proper alias (first for the GUI):

   
openssl.exe pkcs12 -export -in http.crt -inkey http.key -CAfile chain.crt -name http -passout pass:<password> -out http.pfx -chain

3. Repeat for the console proxy:

   
openssl.exe pkcs12 -export -in consoleproxy.crt -inkey consoleproxy.key -CAfile chain.crt -name consoleproxy -passout pass:<password> -out consoleproxy.pfx –chain

4. Now we can import the two PKCS12 keystores into JAVA JCKS keystore with keytool:

   /opt/vmware/vcloud-director/jre/bin/keytool -importkeystore -deststorepass <password> -destkeystore certificates.ks -deststoretype JCEKS -srckeystore http.pfx -srcstoretype PKCS12 -srcstorepass <password>


   
/opt/vmware/vcloud-director/jre/bin/keytool -importkeystore -deststorepass <password> -destkeystore certificates.ks -deststoretype JCEKS -srckeystore consoleproxy.pfx -srcstoretype PKCS12 -srcstorepass <password>


5. We can check if the import was successful:

   
/opt/vmware/vcloud-director/jre/bin/keytool -storetype JCEKS -storepass <password> -keystore certificates.ks –list

6. Now we can import the new certificates to the vCloud Director cell. To do that we need first to stop it:

   service vmware-vcd stop

   Note this will interrupt all running VCD jobs. In order to do graceful shutdown follow http://kb.vmware.com/kb/1033575
   

7. Rerun configuration tool and point to the certificates.ks keystore created in steps 2 and 3. This will import the certificates.

   /opt/vmware/vcloud-director/bin/configure

8. Repeat for the other VCD cells

How can we check the certificates can be properly validated by all clients? The best is to use external certificate checking tool which is not ‘polluted’ by intermediate certificates that might be downloaded via other means. Here are some of them:

http://www.networking4all.com/en/support/tools/site+check/

https://knowledge.geotrust.com/support/knowledge-base/index?page=content&id=SO9557

http://www.sslshopper.com/ssl-checker.html

Another way is to use openssl s_client option:

openssl.exe s_client -connect vcd.domain.com:443

The output should show the whole chain and the root certificate.

You can also try to upload an iso into vCloud Director catalog. The upload actually uses Java script which is not affected by the certificates from Windows store. Note: if you are still running vCloud Director 1.5.0 you can encounter digital signature error of VMware vCloud Director File Transfer Client which expired March 31 2012. This is unrelated to certificate replacement and has been fixed in VCD 1.5.1 where proper time stamp was included in the applet digital signature.

vCenter Certificate Checking

There is also possibility to enforce certificate validation for communication between vCloud Director cells and vCenter(s) or vShield Manager(s). This makes sense as vCloud Director cells are placed in DMZ zone and Man-in-the-middle attack could be used. The descriptions how to do this is in the VMware vCloud Director Security Hardening Guide 1.0). The VCD 1.5 specific version is not public yet (as of May 2012) but the procedure is the same. It requires creation of JCEKS format keystore with CA certificate that was used to sign vCenter and vShield Manager certificates and importing it via VCD – Administration – General GUI. Do not forget to use Subject Alternate Name option when creating vShield Manager certificate to include IP address as well as FQDN.