Are you still on vCloud Director 9.7 (VCD) in multi-cell configuration? Then you are susceptible to Java Message Service (JMS) certificate expiration issue. Read on.
Background
In multi-cell set up VCD cells need to communicate between themselves. They use shared database but for much faster and efficient communication they also use internal ActiveMQ message bus. It is used for activity sharing and vCenter Server events notifications. If the message bus is dysfunctional it slows any operations almost to halt. For this particular certificate issue you will see in the logs similar message:
Could not accept connection from tcp://<primary-cell-IP:port> : javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
In vCloud Director 9.7 the bus communication become encrypted in preparation for other use cases (read here). On upgrade or new deployment of each cell new certificate was issued by internal VCD_CA with 365 day duration. In vCloud Director 10.0 or VMware Cloud Director 10.1 the certificate is regenerated upon upgrade and its duration is extended to 3 years.
To find out the certificates expiry date, run the following command from any cell:
/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates -status
It will for every cell print out its JMS certificate details:
…
Cell with UUID fd0d2ca0-e357-4aae-9a3b-1c1c5d143d05 and IP 192.168.3.12 has jms certificate: [
[
Version: V3
Subject: CN=vcd-node2.vmware.local
Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11
Key: Sun RSA public key, 2048 bits
modulus: 25783371233977292378120920630797680107189459997753528550984552091043966882929535251723319624716621953887829800012972122297123129787471789561707788386606748136996502359020718390547612728910865287660771295203595369999641232667311679842803649012857218342116800466886995068595408721695568494955018577009303299305701023577567636203030094089380853265887935886100721105614590849657516345704143859471289058449674598267919118170969835480316562172830266483561584406559147911791471716017535443255297063175552471807941727578887748405832530327303427058308095740913857590061680873666005329704917078933424596015255346720758701070463
public exponent: 65537
Validity: [From: Wed Jun 12 15:38:11 UTC 2019,
To: Thu Jun 11 15:38:11 UTC 2020]
…
Â
Yes, this particular cell’s certificate will expire Jun 12 2020 – in less than two months!
The Fix
Set a calendar reminder and when the certificate expiration day is approaching run the following command.
/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates --certgen
Or upgrade to vCloud Director 10.0 or newer.
Update 21/05/2020: KB 78964 has been published on this topic. Also if the CA signing certificate is expired you will need to disable SSL altogether, restart the cell, regenerate the cert and re-enable SSL.
Tom Fojta's Blog 
Hello Fojta, to run this renew command I have to stop the cells ? I have to do this command in each cell or just one ? I have to restart the cells ?
Only on one cell, no need to restart.
Sorry, cell restart *is* needed.
Okay, so I run the command in only one cell and restart all my vCloud Cells ?
Correct.
Fojta just one more question I promess, If I update vCloud from 9.7.0.4 to 9.7.0.5, it will correct this issue or not ?
Hi Tomas
Just FYI there seems to be an issue with vCloud 9.7 and 10 where this process creates certificates with an incorrect IP address in the certificate subject alt name (adds the IP address for the cell that you ran the certgen on).
From the support call I have with VMware this is a known issue, apparently fixed in an upcoming patch. The workaround is to either disable JMS SSL (run /opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates –disablessl on one cell and restart all cells) or delete the certificate from the vCloud database then start each cell one at a time.
I don’t see a KB that I can cite as yet.
Jono, thanks for your reply
your solution is correct:
I`ll share required postgress commands to delete JMS certs from DB
select * from truststore where tag = ‘JMS’;
delete from truststore where tag = ‘JMS’;
You should execute this commands with stopped services.
And than bring your cells up one by one.
Hi Tomas, quick question. I want to renew http and consoleproxy certs. The common names for the existing certs are for example http.example.com and consoleproxy.example.com. When running the csr generate command as;
keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass root_passwd -certreq -alias http -file http.csr -ext “san=dns:http.example.com,dns:http”
the csr is generated, however when submitting to public CA for cert creation, the common name is still set to the internal FQDN and private internal domain of the vcd cell. Would you know how i can generate the csr to use the public common name instead by any chance?
Update; unsure if this is the correct method, however i resorted to deleting both untrusted http and consoleproxy certs from certificates.ks –
keytool -delete -alias http -keystore /opt/vmware/vcloud-director/certificates.ks -storepass root_passwd
-storetype JCEKS -v
I than used the old linux command (as per vCD 9 KB) to re-create untrusted certs for http and consoleproxy, which let me specify the CN i wanted to use (my public DNS) –
keytool
-keystore certificates.ks
-alias http
-storepass passwd
-keypass passwd
-storetype JCEKS
-genkeypair
-keyalg RSA
-keysize 2048
-validity 365
-dname “CN=vcd1.example.com, OU=Engineering, O=Example Corp, L=Palo Alto, S=California, C=US”
-ext “san=dns:vcd1.example.com,dns:vcd1,ip:10.100.101.9”
Only after this, did i create the CSR’s again and submit to online public CA, and i no longer got the error ‘Common name invalid’ when renewing the certificates.
You can use “keytool -delete” subcommand to delete a specific certificate from certificate.ks, however you can also recreate a new certificate.ks from scratch. The keystore file is used only during the import of the certs to VCD (via CMT or configure commands) and not used after that.
Hi Tomas
I have a 1 cell vCD (v10.1) for lab on which I had a public certificate that has expired few days ago.
I just realized so generated now self signed certificate with :
./cell-management-tool generate-certs -j -p -o xxx.ks -w yyy
Then imported cert :
/opt/vmware/vcloud-director/bin/cell-management-tool certificates -j -p –keystore xxx.ks –keystore-password yyy
and restarted vCD : service vmware-vcd restart
When connecting to the vCD UI, I can see the new certificate is used. However when accessing the console, it is disconnecting after a few sec, and I have the following message in console-proxy.log
2021-04-20 14:25:49,257 | DEBUG | consoleproxy | ReadOperation | IOException while reading data. Closing transfer. | java.nio.channels.SocketChannel[connected local=/:444 remote=/:39246]
javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:214)
Any idea where the issue is ?
PS: I have also tried to run the commands you recommend in this post with jms:/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates –certgen and restart, but did not help
Thanks
jms certs have nothing to do with console certs. As you generated self signed certs for https and console you need to trust them in your browser. See if https:/vcd.example.com:8443 gives you cert error in the browser.
Hi Tomas
I know it is not related, however as this post was related to certificates and you are very experienced with all vcd versions … i thought you may help.
When i open with the browser the url, i have a warning because it is self signed … but that should be ok, no ?
No. Your client (browser/OS) must trust the certificate explicitly. So it has to be green without any error. The easiest is to add the self signed cert to your trused certificate store
Not sure if this helps; I created a untrusted cert for HTTP and one for consoleproxy, generated a CSR for both using my CA and imported both into java keystore.
I guess as your cert is self signed, you would verify all certs are imported in java keystore;
keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -list
if so, run configure script for network & DB connections
/opt/vmware/vcloud-director/bin/configure
Finally paste the certificate chain in vCD public addresses in PEM format;