Centralized Logging in vCloud Director Environments

I have been researching what the options for centralized logging in vCloud Director environments. Such deployments consist of large number of systems and to have all their logs in one place is the best practice for their additional processing, archiving and troubleshooting. If the logs are scattered in different locations it is very hard to correlate them and apply consistent policies.

I am going to describe the logging configurability and other considerations for following systems from the VMware vCloud stack:

  • vSphere ESXi hosts
  • vCenter Server
  • vCloud Director (cells)
  • vShield (vCloud Network and Security) Edge Gateways
  • vShield (vCloud Network and Security) Manager
  • vCenter Orchestrator

This article is not about what should be used as the central system that collects those logs and how to mine useful information out of them. Syslog compatible system is the essential requirement here however other enterprise features as high availability, log indexing, load balancing, reporting, archiving and extensibility for non-syslog sources should be considered.

vSphere ESXi Hosts

Log collection options on ESXi hosts are the most advanced out of all considered components. ESX logs are by default stored locally either on local datastore or in memory. They can be redirected to any other datastore however only useful option in large deployments is to redirect them to syslog server. This is done by changing the Syslog.global.logHost variable. Here we can put multiple destination syslog servers. We can also choose the protocol (TCP/UDP/ssl) and port number. The logs are then sent over vmkernel management network.

Note: ESXi produces large number of logs (vmkernel, hostd, vpxa, fdm, …) which are stored in separate files but combined into one syslog stream. Hostd and vpxa logs are set by default to verbose level and are quite chatty, so I usually change them to info level. This can be done by changing Vpx.vpxa.config.log and Config.HostAgent.log.level advanced variables.

vCenter Server

The Windows installable vCenter Server unfortunately cannot be easily configured to send its logs to an external syslog server. Therefore additional agent must be installed. The vCenter Server Appliance (vCSA) edition which runs on linux OS on the other hand utilizes syslog-ng which can be easily reconfigured to forward its logs to remote syslog. William Lam wrote excellent post how to do that here.

vCSA 5.1 when used with the internal PostgreSQL DB is currently supported while managing only up to 4 ESX hosts which might be enough for management cluster. For the vCloud resource clusters either vCSA with external Oracle database or Windows installable vCenter Server must be used.

vCloud Director

vCloud Director cells produce number of logs. There is an audit log which persists in the database for 90 days. It can be forwarded to an external syslog server by changing the $VCLOUD_HOME/etc/response.properties file on each vCloud Director cell. However only one syslog target can be defined with UDP/514 port. Therefore it is recommended to redirect this log to local RHEL syslog (127.0.0.1) and then use its forwarding features. The detail description how to do this is in vCAT 2.0 Public VMware vCloud Implementation Example which can be downloaded from here.

Additional troubleshooting and API access logs are stored locally in $VCLOUD_HOME/logs directory. These logs are produced by Apache log4j logging utility and can be forwarded by creating additional appender in the $VCLOUD_HOME/etc/log4j.properties directory. This is described in the KB article 2004519.

EDIT 1/10/2013: A new KB article has been posted about editing log4j.properties file: KB article 2004564.

vShield Edge Gateways

vShield Edge Gateway is multi-interface vShield Edge virtual device that connects vCloud Director organization VDC networks to external networks. These Edge devices are deployed as needed by vCloud Director through vShield Manager. vCloud Director holds all their configuration details which means that if their properties are edited directly through vShield Manager, vCloud Director is not aware of those changes and could revert them back. Why am I mentioning this? Each Edge device can be configured to send its logs to two remote Syslog servers. Additionally protocol (UDP/TCP) can be specified as well. However vCloud Director only allows to set up two default syslog servers (System > Administration > System Settings > General > Networking). And these will be applied to every Edge device deployed (either Edge Gateways or vApp Edges). So no per tenant or per network configuration is out-of-the-box possible (unless orchestrated with blocking tasks and vCenter Orchestrator).

Multi-interface Edge Gateway allows us to create dedicated logging external network on which the Syslog server (or forwarder to the Syslog server) can be placed. If we would want to collect logs also from vApp Edges the tenant would have to make sure that the vApp Edge can route the syslog traffic through the Edge Gateway to the Syslog server. If it will not (or cannot because it is isolated from Edge Gateway) then the vApp Edge logs are lost.

The provider could decide that he wants to collect only Edge Gateway logs and prohibit any other devices (vApp Edges or any other VM) to send the logs to provider’s Syslog server. This can be accomplished with design shown in the following picture:

Edge GW Logging

Each Edge Gateway has two external networks. One connected to the internet with Sub-allocate IP Pool with external IP addresses which the customer can use for Network Address Translation (NAT) to enable internet access to or from his VMs. The second external interface is connected to the logging external network. Customer cannot create any NAT rules because Sub-allocate IP Pool on that network is not populated. To block routed traffic from VMs behind the Edge Gateway firewall rule “allow only logging external network” is created (allow 10.0.5.0/24).

vShield Manager

vShield Manager (vSM) produces audit and system event logs. These can be forwarded to only one remote syslog server which can be configured in the vSM Settings&Reports > Configuration > General menu. Additionally port can be configured as well, however only UDP protocol is used.

vCenter Orchestrator

Similarly to vCenter Server, vCenter Orchestrator comes in two flavors – Windows installable or virtual appliance. KB article 1010956 describes the location of the files in the Windows installable edition. The same logging redirection can be used as described in the vCenter Server section above.

Note: This articles was written for the following versions: vSphere 5.1, vCloud Director 5.1, vCloud Network And Security 5.1 and vCenter Orchestrator 5.1.

Advertisements

4 thoughts on “Centralized Logging in vCloud Director Environments

  1. Excellent. I’ve been trying to find more information on this for a while.
    Do you have any values you can apply to each of these for approximate logsize requirements?
    Its very difficult to design a sizing requirement for a syslog service capability when there are no starting figures anywhere that I can find.

  2. Hi Tomas, just one point of clarification on the vCD cell audit logs. If you edit the $VCLOUD_HOME/etc/responses.properties file on a host that is already configured it will do nothing unless you re-run the configuration script on that host specifying -r $VCLOUD_HOME/etc.responses.properties.

    If you want to modify the audit syslog server without re-running the configuration script you can edit the $VCLOUD_HOME/etc/global.properties file, and then restart the services on the cell.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s