Multitenant Logging with VMware Cloud Director

Topic of multitenant logging in VMware Cloud Director (VCD) especially with NSX-T came up recently in multiple conversations, so I will summarize the capabilities and options as they are today.

Audit and Event Logs

Information about VCD events such as user logins, vApp operation, VM power-on etc are available in the Audit and Event log. Tenant can access their particular logs via API – AuditTrail Events and Tasks (type=task query) or directly in the UI.

Event Log UI

An additional option is to receive events and notifications via MQTT message bus as was described here.

NSX-V Networking Logs

With NSX-V each tenant would have one or more Org VDC Gateways each backed by its own NSX-V Edge Service Gateway (single or two VMs in HA mode). The tenant could configure their own syslog endpoint to send all Edge logs there. As each Edge could have two syslog entries, the provider could set up their syslog endpoint as well to receive the same logs.

Tenant Edge Gateway Syslog Configuration

Provider Edge Gateway Syslog Configuration

And this is how the configuration looks in NSX-V:

Distributed firewall logs are generated in vmkernel of each ESXi host. It means they are sent to syslog configured at ESXi level which is not multitenant. However VCD configures a rule tag which is unique for each Org VDC (its UUID) so it is possible to filter logs for a specific tenant.

Org VDC UUID rule tag configured for each DFW entry in NSX-V

NSX-T Networking Logs

Org VDC gateways in NSX-T Org VDCs are backed by NSX-T Tier-1 gateways which are logical entities on shared NSX-T Edge Nodes. This means it is not possible to configure tenant’s syslog on them as was done with NSX-V. Therefore VCD will configure Log label for each GW firewall entry. The log is composit of first 10 characters of the UUID of the GW parent object (Org VDC or Data Center Group), 5 characters of the GW name and 5 characters of the GW UUID.

Note that you need at least NSX-T 3.2 and VCD 10.4.

For distributed firewall the situation is identical to NSX-V. The only difference it that the log label is composit of first 5 characters of the Data Center Group name and 27 characters of its UUID.

NSX-T DFW Log Label

Additional improvements are planned in NSX-T 4.1+ to allow multitenant access to other log types.

Log Filtering and Forwarding

To test the functionality vRealize Log Insight supports up to 10 log forwarders. These can be configured in the Log Management section. In the screenshot below you can see Distributed Firewall for NSX-V and NSX-T and Gateway Firewall for NSX-T forwarding to tenant syslog 10.0.2.29 configured.

vRealize Log Insight filtering and forwarding

vRealize Operations Management Pack for NSX-V and Log Insight Integration

Quick post about an issue I discovered in my lab during upgrade to NSX 6.3.3. This particular NSX version has a silent new feature that verifies if syslog configuration on Edges is correct. If the syslog entry is incorrect (it is not an IP address or FQDN with at least one dot character or does not have TCP/UDP protocol specified) it will not let you save it. This however also means that older Edges (with version 6.3.2 or older) that have incorrect syslog setting will fail to be upgraded as the incorrect config will not be accepted.

So how does it relate to the title of the article? If you have vROps in your environment with NSX-V management pack and you have enabled Log Insight integration, the Management Pack will configure syslog on all NSX components. Unfortunately in my case it configures them incorrectly with only hostname and no protocol. This reconfiguration happens roughly every hour. This might be especially annoying in vCloud Director environment where all the Edges are initially deployed with syslog setting specified by VCD, but then are changed within an hour by vROps to something different.

Anyway, the remediation is simple. Disable the Log Insight integration of the vROps NSX Management Pack as shown on the picture below.

Collect vCloud Director Cell Logs with Log Insight Agent

While it is possible to redirect vCloud Director cell logs by editing log4j.properties file to remote syslog server (see KB 2004564) there is an alternative agent based method utilizing vRealize Log Insight.

Log Insight agent is installed on each cell and then remotely managed from Log Insight server. Here are some advantages of this approach:

• no manual edits of log4j file which gets overwritten with each upgrade
• as we do not rely log4j logger we are able to collect also API request log files which are generated by Jetty
• agent uses reliable TCP communication as opposed to unreliable UDP
• we no longer rely on source IP to identify sender; cells can use source NAT (with single IP) to communicate with Log Insight server and we can still distinguish them
• we can remotely change which logs we want to monitor (info vs debug)
• and much more

Here is quick configuration how to:

1. Download Log Insight Agent from Log Insight Server. It is already customized installation for your vRLI server. Administration > Agents > scroll down > Download Log Insight Agent Version 3.6.0 > pick rpm package
2. Upload rpm file to each cell and install it with rpm -i VMware-Log-Insight-Agent-3.6.0-4148343.noarch_XXX.rpm
3. Back in Agents configuration create active agent group from vCloud Director Cell Server template (copy template icon)
4. Create hostname filter (use ? for any character substitution, you can add multiple entries in one line for ‘logical or’ or multiple lines for ‘logical and’
5. Optionally edit agent configuration to include additional files or directories

Edit 11/30/2017:
Example of Agent Configuration based on vCloud Director 9.0

[filelog|vcd-essential]
directory=/opt/vmware/vcloud-director/logs
include=vcloud-container-debug*;upgrade*;vmware-vcd-support*;watchdog*;console-proxy*;statsfeeder*;server-group-communication*;cell-runtime*
event_marker=(\d{2}|\d{4})-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s
tags={"vmw_product":"vcd"}

[filelog|vcd-API]
directory=/opt/vmware/vcloud-director/logs
include=*request.log*
event_marker=\b(?:\d{1,3}\.){3}\d{1,3}\b
tags={"vmw_product":"vcd"}

Edit 2/2/2021
Example for VMware Cloud Director 10.2 appliance

[filelog|vcd]
directory=/opt/vmware/vcloud-director/logs
include=vcloud-container-debug*;upgrade*;vmware-vcd-support*;watchdog*;vcloud-container-info*;cell*;request.log*;cell-management-tool.log*;cell-runtime.log*;cell.log*;cloud-proxy.log*;queries*;networking.log*;server-group-communications.log*;upgrade-*;service-wiring.log*;statsfeeder.log*;networking-wire.log*
event_marker=(\d{2}|\d{4})-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s
tags={"vmw_product":"vcd"}

[filelog|vcd-API]
directory=/opt/vmware/vcloud-director/logs
include=*request.log*
event_marker=\b(?:\d{1,3}\.){3}\d{1,3}\b
tags={"vmw_product":"vcd"}

[filelog|vcd-appliance]
directory=/opt/vmware/var/log/vcd
include=*.log*
tags={"vmw_product":"vcd-appliance"}

[filelog|vcd-vpostgres]
directory=/var/vmware/vpostgres/10/pgdata/log
include=*.log*
tags={"vmw_product":"vcd-vpostgres"}