vCloud Director with TLS-only Connection with External Database

Very brief blog post to document how to install vCloud Director with external database that does not support plain text connections.

In general the process is to do the initial set up with plain text DB connection and then switch to TLS – see the official docs here. That will however not work if the external database supports only TLS connection.

Instead this process must be used:

1. Import DB certificate (unless it is publicly signed) to cell default JRE keystore.
2. Use unattended configuration.

Example:

# /opt/vmware/vcloud-director/jre/bin/keytool --import -trustcacerts -keystore /opt/vmware/vcloud-director/jre/lib/security/cacerts -alias psql -file /opt/vmware/vcloud-director/etc/psql.crt

 

Enter keystore password: changeit

 

Owner: CN=10.0.4.64

Issuer: CN=10.0.4.64

Serial number: cb64ae0954184182

Valid from: Fri Nov 22 14:10:39 GMT 2019 until: Sat Nov 21 14:10:39 GMT 2020 Certificate fingerprints:

MD5:  04:4F:8F:C5:9C:CC:D5:E8:F1:50:C1:85:51:D4:FB:AD

SHA1: 22:53:FF:71:A7:EC:9B:D1:74:79:D5:95:46:71:F6:38:A7:E7:F8:4E

SHA256: 08:7C:27:B4:FB:32:04:DE:AF:BB:FE:9D:47:1D:38:46:C8:F4:7C:69:73:DE:8D:CB:BD:2A:A5:B2:11:12:68:DD

Signature algorithm name: SHA256withRSA

Subject Public Key Algorithm: 2048-bit RSA key

Version: 3

 

Extensions:

 

#1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

 

#2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[

CA:true

PathLen:2147483647

]

 

#3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

 

Trust this certificate? [no]:  yes

Certificate was added to keystore

 

 

# /opt/vmware/vcloud-director/bin/configure --unattended -dbhost <DB IP address> -dbname vcloud -dbpassword vcloud -dbtype postgres -dbuser vcloud --database-ssl true –dbport 5423 -ip <cell-ip> --primary-port-http 80 --primary-port-https 443 -cons <cell-ip> --console-proxy-port-https 8443 -k /opt/vmware/vcloud-director/etc/certificates.ks -w <keystore password> -g

.......................................\

Database configuration complete.

 

# /opt/vmware/vcloud-director/bin/cell-management-tool system-setup --email admin@vcloud.com --full-name 'System Admin' --installation-id 33 --password 'VMware1!' -system-name vcd --unattended --user administrator

Creating admin user.

Setting system details.

Completing system setup.

System setup is complete.

PostgreSQL: Beware of the bloat!

During the last weekend my vCloud Director lab died. And the reason was PostgreSQL DB filled up all the disk space. How could that happen in my small lab with one running vApp?

PostgreSQL database when updating rows actually creates new ones and does not immediately delete the (now dead) old rows. That is done in a separate process called vacuuming.

vCloud Director has one pretty busy table named activity_parameters that is continuously updated. And as you can see from the below screenshot (as reported by pgAdmin table statistics) the table size is 26 MB but it is actually taking 24 GB of hard disk space due to the dead rows.

Another quick way to check DB size via psql CLI is:

\c vcloud
SELECT pg_size_pretty (pg_total_relation_size(‘activity_parameters’));

Vacuuming takes times and therefore it can be tuned in postgresql.conf via a few parameters which VMware documents specifically for vCloud Director here or here. Make sure you apply them (I did not). Another issue that could prevent vacuuming to happen is a stale long running transaction on the table.

The fix:

• short term: add more disk space
• long term: make sure postgresql.conf is properly configured
  autovacuum = on
  track_counts = on
  autovacuum_max_workers = 3
  autovacuum_naptime = 1min
  autovacuum_vacuum_cost_limit = 2400
• manually vacuum the activity_parameters table with the following psql CLI command:
  VACUUM VERBOSE ANALYSE activity_parameters;

And do not forget to monitor free disk space on your PostgreSQL host.

 

Upgrade PostgreSQL version 9 to 10

I had to perform multiple PostgreSQL database upgrades from version 9 to version 10. The database was used for vCloud Director but I believe it is generic enough for other purposes.

The base operating systems I am using is CentOS 7.

Here follows the step-by-step procedure:

1. Create database backup:
   su – postgres
   pg_dumpall > /tmp/pg9backup
   exit
2. Shutdown and uninstall old PostgreSQL v9:
   systemctl stop postgresql-9.5.service
   yum remove postgresql*
3. Archive old datafiles (you will need them later):
   mv /var/lib/pgsql/data/ /data.old
4. Install new PostgreSQL v10:
   yum -y install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
   yum -y install postgresql10-server
   systemctl enable postgresql-10
5. Initiate and start DB:
   su – postgres
   /usr/pgsql-10/bin/initdb
   cp /data.old/pg_hba.conf /var/lib/pgsql/10/data/
   cp /data.old/postgresql.conf /var/lib/pgsql/10/data/
   exit
   systemctl start postgresql-10
   
6. Restore backup
   su – postgres
   psql -d postgres -f /tmp/pg9backup
7. Reboot server. If everything works, you can delete your pg9backup and data.old archive.