vCloud Connector 2.0 Observations

I have been playing with vCloud Connector 2.0 which is already a third release of the tool that enables VM (vApp) transfers between various VMware clouds based on vSphere or vCloud Director. Here follows a bunch of notes that I came up with that might help others. Note I expect that the reader is familiar with basic functionality and architecture of vCC.

Compatibility

vCloud Connector 2.0 (vCC) is backward compatible with vCloud Director 1.5, however some advanced feature like Content Sync will work in such deployments with limitations (modified templates are not removed, but added instead with timestamp in the name) or will not work at all (Stretch Deploy). vCloud Connector 1.5 does not work with vCloud Director 5.1.

Architecture and Network Flows

The architecture is similar to vCC 1.5, but the ports have changed (port 8443 not needed anymore). Here is the picture from the Installing and Configuring vCC 2.0 guide and let me dive deeper into some of the network flows (black circled numbers).

Flow 1

Although vCC Server has a web interface (VAMI) available at port 5480, the interface can be used only for appliance configuration and setting up connectivity to vCC Nodes. Internet Explorer is recommended to use here as Firefox or Chrome did not display some VAMI interface parts properly. The vCC Advanced Edition license key is entered here and SSL certificates related to the vCC Client – vCC Server communications can be enabled or generated and uploaded here as well. VAMI interface always uses https (on port 5480) with self signed certificate by default that cannot be replaced via the GUI, but if required it can be replaced in the following file on the appliance:

/opt/vmware/etc/lighttpd/server.pem

vCC end-users will use either vSphere client (the .NET version, as web client is not supported yet) or vcloud.vmware.com portal for actual management of the vApp transfers and other vCC features.

Although the picture shows port 80, if SSL is enabled, 443 will be used instead. If the replaced certificates do not use intermediate CA, the web interface cannot be used for their import and java keytool command must be used instead from appliance CLI as described in the installation guide. For some reason I was not able to create the private key with the GUI, but java keytool did the job.

Flow 2

vCC Server needs to be able to reach all vCC Nodes so the arrow should be also between vCC Server and the vCC Node in the destination cloud. Again the communication can go over port 80 or 443 depending on SSL configuration – this time on the vCC Nodes. The same caveat as with Server applies when replacing the certificates.

Note: Enabling SSL is recommended here as discussed below.

Flows 3 and 6/7

Prior attaching vCC Nodes to vCC Server, the Nodes need to be configured to connect to a cloud (vCenter or vCloud Director). SSL (port 443) is always used, but if you do not want to enable the Ignore SSL Cert checkbox, vCenter or vCloud need to have CA signed certificates. If you are using enterprise CA, you have to import the CA root certificate to a different keystore than the one available in the GUI as described in KB 2045007.

Flow 5

For the inter node communication, one node is designated as the Controller. The Controller initiates the connection. Which node is picked as the Controller depends on the Public checkbox setting in the vCC Node registration at vCC Server.

Public vCC Node

It is expected that the non Public vCC node is unreachable from outside and therefore has to be the initiator of the communication between nodes and is therefore the Controller. So the Controller Node works either in push or pull mode. If the other Node has SSL enabled (see Flow 2), https port 443 is used for the transfer between nodes. If the other Node does not have SSL enabled the transfer will fail.

Shared Node

This is a new feature for vCloud Director deployments when the provider can deploy shared (multitenant) node and connect it to provider’s cloud. The tenants then do not have to deploy their own nodes inside their organization VDCs with all the troubles of securing connectivity and appropriate transfer storage.

In highly secure public clouds it gets tricky where and how to deploy the shared node. It cannot be load balanced, but provider can deploy multiple shared nodes and give their IP addresses only to specific groups of tenants. The node should be as close as possible to the vCloud API somewhere in DMZ accessible from the internet but if Web Application Firewall is used it should still be in-between the node and API as it could be used as an attack vector to other organizations. The node does not keep any vCloud credentials for communication with vCloud API. Those are transferred by vCC Server so again SSL should be enabled (see Flow 2).

Content Sync

This is a new feature, great for maintenance of catalogs in various clouds or might be also used by the provider for management of public catalog directly from vSphere where a vSphere folder is automatically synced with a vCloud Director catalog. Note however that vCC Advanced Edition license is needed which currently cannot be obtain with provider VSPP license, but only through vCloud Suite bundles.

The default polling interval for synchronization is 6 hours. It can be changed but the change is unsupported. Following file can be edited on the vCC Server:

/usr/local/tcserver/vfabric-tc-server-standard/server/webapps/agent/WEB-INF/spring/appServlet/task.xml

Look for <property name=”jobExecutionIntervalInMinutes” value=”360″ />.

Although the documentation states that ports 8443 and 8080 are used for Content Sync, my understanding is that they are used only internally on the vCC Server.

Stretch Deploy

This is again a new but licensed feature which enables migration of VMs between clouds without needing to change their IPs or MAC addresses thanks to a VPN connection (not VXLAN as often confused) which is established between the original and destination cloud. The SSL VPN is established between Edges on the vApp networks so this is quite different from the Site-to-Site IPSec VPN between Edge Gateways even though it shows up in vShield Manager in the IPSec VPN section. Its configuration is not exposed in the vCloud Director GUI at all.

Stretch Deploy Site-to-Site SSL VPN

There is quite a long list of prerequisites to get this working and some of them are out of tenant’s controls as they relate to the provider’s vCloud Director architecture – e.g. vSphere and vCloud Network and Security versions and type of distributed virtual switch used. Stretch Deploy will not work with Cisco Nexus 1000V switch. The tenant most likely will not know which switch the provider is using until he will experience following error:

Unable to update network “Stretched_VM_network”.
java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (100): A specified parameter was not correct.
selectionSet.dvsUuid

The source and transferred destination VMs need to be connected to vSphere Distributed Switch 5.1 as the unclaimed traffic needs to be sent over the SSL tunnel and this is not currently supported with other virtual switches.

The Stretch Deploy process is relatively complicated with many actions that are happening in the background. This is all abstracted from the user as he can start the process easily from vCC GUI. However if you want to end the stretch deploy by removing the remote VM, or bring it home back this must be done manually.

Stretch Deploy activities as seen by the vCenter (note both source and destination clouds were managed by the same vCenter)

Delete Stretch Deployed VM

• stop the remote vApp, this will terminate the VPN connection and destroy the vApp Edge
• delete the remote vApp
• delete the IPSec configuration in the vSphere cloud Edge in vShield Manager
• delete vCenter custom attributes of the VM which was stretched deployed (DatacenterExtendedEntityId, DatacenterExtensionRole)

Bring Home the Stretch Deployed VM

This must be done by running a script from the vCC Node managing the private cloud. So an access to the Node is needed, the script needs to be untared and quite a lot of information must be typed in when executed and their correctness is not verified until they are all typed in. This is definitely not task for an average user and I expect this part might be improved in later releases.

Graceful Shutdown of vCloud Director Cell

I have been challenged by one of my customers how to properly shutdown vCloud Director cell without any disruption of the service if multiple cells are used. Although we have KB article 2034994 about this particular subject it omits some important details.

When vCenter is connected to vCloud Director a VC Proxy service is started on one of the cells. The service is responsible for monitoring of active vCenter tasks and inventory updates which are then shared with other cells. Unless there is a network partition between the cells there is always one vCenter proxy service for one vCenter. Multiple VC Proxies can run on one cell. You can see which cell is running the VC Proxy service at the vCenter screen in the vCloud Director Admin interface.

The screenshot shows two vCenters connected to vCloud Director with one having its vCenter proxy on vcloud1 cell and the second on vcloud2 cell.

If the VC Proxy service is not running most of the activities in the vCloud Director that require vCenter will not work properly. For example simple creation of a vApp with one VM will fail with message:

Folder vApp_system_34 (8ce90b57-da8b-4714-914b-5073457155b0) does not exist in our inventory, but vCenter Server claims that it does.

This is because the inventory listener on the VC Proxy was not running and vCloud Director could not verify successful creation of vApp folder in vCenter. When a cell with VC Proxy service dies the service fails over to a surviving cell. However that failover takes 5 minutes which is govern by vcloud:vcloud.heartbeat.failoverTimeoutMsecs property (stored in vCloud Director database). I am not aware if it is supported to change this value.

Anyway in order to shutdown a cell gracefully we need to move the VC Proxy service to another cell. This can be done by simple reconnect of vCenter and the move is very quick without any disruption of the running tasks.

Reconnect vCenter

I have observed that if possible different cell then the original and the least loaded (in terms of number of VC Proxy services) is chosen. This is also good for manually distributing the load if there are multiple vCenters and multiple cells (good practice is to have at least N+1 cells, where N is number of vCenters).

So what should be the correct graceful cell shutdown procedure?

1. Make sure the cell is not running any VC Proxy service. No checkmark should be in the vCenter column of the Cloud Cells inventory in the vCloud Director Admin interface.If yes, then reconnect vCenters that have VC Proxy running on the cell.
2. Quiesce the cell with the cell-management-tool:

   $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –quiesce true

   where <user> is vCloud administrator username

3. Monitor the number of outstanding active tasks on the cell and wait until it reaches 0.

   $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –status
   Job count = 0
   Is Active = false

4. Shutdown the cell. This can be done also with cell-management-tool. What I noticed is that it takes multiple attempts (usually two), as the first time only the watchdog service is terminated.

   # $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –shutdown

   # service vmware-vcd status

   vmware-vcd-watchdog is not running
   vmware-vcd-cell is running

   # $VCLOUD_HOME/bin/cell-management-tool -u <user> cell –shutdown
   # service vmware-vcd status

   vmware-vcd-watchdog is not running
   vmware-vcd-cell is not running

Rate Limiting of External Networks in vCloud Director and Nexus 1000V

There is a new feature in vCloud Director 5.1 which was requested a lot by service providers – configurable limits on routed external networks (for example Internet) for each tenant. Limits can be set both for incoming and outgoing directions by vCloud Administrator on tenant’s Edge Gateway.

Edge Rate Limit Configuration

However this feature only works with VMware vSphere distributed switch – it does not work with Cisco Nexus 1000V or VMware standard switch. Why? Although the feature is provided by the Edge Gateway, what is actually happening in the background is that vShield Manager instructs vCenter to create a traffic shaping policy on the distributed vswitch port used by the Edge VM.

vSphere Distributed Switch Traffic Shaping

Standard switch does not allow port specific traffic shaping and Nexus 1000V management plane (Virtual Supervisor Module) is not accessible by the vShield Manager/vCenter. The rate limit could be applied on the port of the Cisco switch manually, however any Edge redeploy operation, which is accessible by the tenant via GUI would deploy a new Edge and use different port on the virtual switch and tenant could thus easily disable the limit.

For the standard switch backed external network vCloud Director GUI will not even present the option to set the rate limit, however for the Nexus backed external network the operation will fail with similar error:

Cannot update edge gateway “ACME_GW”
java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10086): Traffic shaping policy can be set only for a Vnic connected to a vmware distributed virtual portgroup configured with static port binding. Invalid portgroup ‘dvportgroup-9781′.

Nexus 1000V Error

Btw the rate limit can be set on the Edge (when not using vCloud Director) also via vShield Manager or its API – it is called Traffic Shaping Policy and configurable in the vSM > Edge > Configure > Interfaces > Actions menu.

vShield Manager Traffic Shaping

Do not forget to consider this when designing vCloud Director environments and choosing the virtual switch technology.

How to Change vCloud Director GUI Language

Just a brief reminder mostly for myself how to change vCloud Director GUI Language without changing the web browser language.

Just append /?locale=xx_XX at the end of the URL, where xx_XX is derived from the following supported languages:

• English: /?locale=en_US
• German: /?locale=de_DE
• French: /?locale=fr_FR
• Japanese: /?locale=ja_JP
• Korean: /?locale=ko_KR
• Chinese: /?locale=zh_CN

Example: https://vcloud.fojta.com/cloud/org/tom/?locale=ja_JP

vCloud Director: Online Migration of Virtual Data Center

vCloud Director completely abstracts underlying virtual resources from the consumers who get compute and storage resources represented by virtual datacenters (VDC) with given tiered profile (e.g. gold – silver – bronze). However the provider must care about the actual physical hardware and from time to time is facing the issues of upgrades and migrations.

Fortunately it is possible to migrate whole Provider VDCs non disruptively from the old hardware to a new one with no or minimal impact on the vCloud customers with no downtime or their VMs running in the cloud.

vCloud Director 5.1 has two features that help to accomplish this: elastic VDC (VDCs spanning multiple vSphere clusters) and merging of Provider VDCs. I already wrote about elastic VDCs in the post about Allocation Pool changes so please read that article first if that concept is new to you.

The online migration process from the old to the new hardware in high level works like this:

• Let’s say that the Provider VDC (PVDC) called GoldVDC is backed by Cluster1 consisting of old hardware.
• A new Cluster2 is created with new hardware.
• A new PVDC is created – let us call it GoldVDCnew from the Cluster2 and merged with Cluster1. Although we could add the new Cluster2 directly to the GoldVDC this would not allow us to retire the old hardware as it is not possible to detach the primary resource pool from a PVDC.
  
• We can now rename PVDC GoldVDCnew to GoldVDC and disable Cluster1. This has no impact on running VMs however any newly deployed or power on VMs are already placed to Cluster2.
  
• Now we have to migrate all the workloads from Cluster1 to Cluster2 and then detach the Cluster1 from the PVDC.
  

The actual migration between Clusters (or resource pools) has to take into account following 5 resources that exist in the VDCs – vApps, catalog templates, catalog images, Edge Gateways and vApp Edges.

vApps

vCloud Director actually does not use vSphere vApp objects. vCloud vApps are from the point of view of vSphere infrastructure just a collection of VMs. So we just need to migrate the VMs. This cannot be done from within vSphere because vCloud Director keeps track in which resource pool each VM is placed. Additionally vCloud Director also needs to apply proper resource pool reservations and limits based on the org VDC allocation type. There is however migrate option in vCloud Director that can be used. This can be done from GUI or with API (see the end of this article). Note: the migration leverages vMotion with shared storage. It is not possible to migrate this way between clusters without shared storage even though vSphere 5.1 has so called Enhanced vMotion (aka shared nothing vMotion).

Catalog Templates

Migration of catalog templates is more difficult. Again the vCloud template is quite different from the vSphere template. vCloud templates are basically powered off VMs. Although migration at the vSphere level seems not as harmful as in the previous case, because no resource pool settings must be configured (catalog VMs are never powerd on), we would still encounter a problem when we would try to detach Cluster1 as vCloud Director keeps track of VM to Resource Pool associations.

Unfortunately the GUI migration process from the step 1 cannot be used. The GUI workaround is to open each catalog and move each catalog VM to the same catalog. This basically creates a clone of the VM which gets registered to a Cluster2 host and the original VM is deleted. This is however very expensive operation from the storage perspective. The cloning operation needs temporarily extra space and creates quite a lot of I/O storage traffic. Fast provisioning (linked clones) can help here.

The second alternative is to use the same API call as in the first case. Although this is not documented it works (see the example at the end of the article).

Catalog Media

ISO or FLP images are stored on vSphere datastores in special folder <VCD Instance Name>/media/<org ID>/<org VDC ID>/media-<media ID>.<ISO|FLP>. vSphere (vCenter) does not keep track of these objects in any way. vCloud Director stores the datastore moref and the media folders in its database. The media upload is done by the vCloud Director Cells with NFC – VMware Network File Copy protocol via any ESX host that is connected to the datastore. Therefore as long as the Cluster2 has access to the media datastores nothing needs to be migrated.

vShield Edges

Gateway and vApp Edges are always running VMs placed in System vDC resource pool in Cluster1. If a vApp with routed vApp network is powered off the particular vApp Edge is destroyed. When the vApp is started again a new vApp Edge is deployed with identical configuration and would be placed into the new Cluster2. Simple vMotion between cluster seems to work at first but is definitely not recommended. vShield Manager keeps track to which cluster is each Edge deployed. Any major Edge configuration change (upgrade to new version or upgrade ot full configuration) would try to deploy the Edge to the original cluster.

Edge redeploy is an operation with minimum impact on the network flows going through the virtual router. A new Edge VM is deployed by vShield Manager, identical configuration is pushed to it and then networks are disconnected from the old Edge and connected to the new one. This might have impact on loosing an IPsec VPN connection or load balanced session otherwise the disruption is minimal. The Edge redeploy however cannot be done directly from vShield Manager (too bad as there is a nice script for this: see KB 2035939) because vShield Manager knows nothing about restrictions made in vCloud Director on the PVDC (the disabled Cluster1). Edge Gateway redeploy and routed/fenced network reset must be done from vCloud Director. This can be done from the GUI (however it is not trivial to find all the running vApp Edges) or with vCloud API.

Other Considerations

There are some limitations or considerations that need to be taken into account:

• VDC elasticity currently (version 5.1) works only within vCenter Datacenter domain and all clusters need to use the same distributed switch for external networks and network pool.
• Reservation allocation type Org VDCs do not currently support elasticity of VDC (those workloads cannot be migrated).
• Both clusters should have access to the same storage. If storage migration is required do it as independent second step.
• vSphere vMotion restrictions apply: if the new hardware has newer generation CPU leverage EVC and lower the compatibility of the new cluster to the old hardware. Once the old hardware is retired the EVC mode can be changed and any restarted (full power cycle required) or new VMs can take advantage of it. Obviously migrations between different CPU architectures is not possible (AMD vs Intel),
• 1 GHz of old CPU is not equal to 1 GHz of a newer generation CPU. Therefore do not mix them in elastic VDCs unless for above mentioned migration reasons. This could also impact Chargeback – customer will get different (higher) performance for the same cost.

vCloud API Examples

As mentioned above vCloud VMs can and vCloud template should be migrated with an API call. The request looks like this:
POST API-URL/admin/extension/resourcePool/id/action/migrateVms with Request body containing MigrateParams.

VM migration example:

Template migration example

vCenter Chargeback and vCloud Director 5.1 in DMZ

Recently in the Load Balancing vCloud Director Cells with vShield Edge post I described typical vCloud Director architecture where vCloud Director Cells are multihomed and placed in DMZ with firewall separating it from the internet and the management zone.

When vCloud Director is installed two IP addresses have to be specified. One for binding the GUI portal and API access the other for binding the remote console proxy. In the diagram above there are both on the nortbound interface(s) and load balanced and firewalled from the internet. The cells need to communicate with vCloud Director database, Resource Group ESX hosts and other management systems (Resource Group vCenter, vShield Manger, etc.). For this in the diagram above the southbound network interface is used. Btw it does not need to be specified during the vCloud Director installation as guest OS IP routing table decides which interface is used. The cells also need to talk to each other directly (see the note at the end of the article). Some customers do not allow multihomed VMs that would bridge security zones – then the set up looks different but that is out of scope for this article..

So what all that has to do with Chargeback? vCenter Chargeback Manager is additional management system (usually installed in the management zone) that needs to communicate with vCloud Director in order to retrieve grouping of tenant resources (resource pools, VMs, networks, etc.) to so called hierarchies. Chargeback uses vCloud Director Data Collector for this. The collector can be installed together with the Chargeback server or as a standalone component. In the vCloud Director 1.x version the VCD Data Collector directly retrieved the needed information from vCloud Director database. The VCD Database has usually direct connectivity to the management zone so that was not an issue.

However in vCloud Director 5.1 the Data Collector does not talk to the VCD Database anymore and instead uses vCloud API (bound to the north VCD cell network interfaces) to retrieve the hierarchies. And this might be an issue if security rules prohibit routing between management zone and the internet DMZ zone. Note besides Chargeback there could also be deployed additional systems that need to talk to vCloud API – vCenter Orchestrator, vFabric Application Director, vCloud Automation Center or any third party system.

There are two options how to solve this problem:

1. Placement into DMZ

We can keep the Chargeback in the management zone, but deploy the Chargeback VCD Data Collector into DMZ next to the cells. The following diagram shows the setup.

The data collector is a dual homed Windows VM. It talks to the vCloud API via its north interface – I have included additional vShield Edge internal IP address (10.0.1.2) for the internal load balancing of cell API interfaces (10.0.1.60 and 10.0.1.62). The south interface is used for communication with the Chargeback database.

2. vCloud Director Management Cell

Additional vCloud Director cell(s) is deployed with the http (GUI + API) interface bound to the south interface.

The VMware Remote Console (VMRC) proxy is still bound to the north network interface and can be used for load balancing VMRC traffic coming from the internet. All the management systems that need to talk to vCloud API can do so directly from the management zone by talking to the 10.0.4.64 IP address.

Note: As was said at the beginning of the article all the vCloud Director cells communicate with each other over ActiveMQ message bus. The bus is always using the primary (http) IP address of the cell and this cannot be changed. Therefore each cell must be able to reach all the others on the http interface. So a design where the VCD3 (API) cell is connected only to the management network would not work. What would be the impact? For example there is always only one vCenter Proxy services running on one of the cells for a particular resource vCenter Server. The VC Proxy service is responsible for invoking operations on the vCenter and listening back on their progress. However any cell can pick up a particular task and if it cannot reach the VC Proxy the task will fail.

Centralized Logging in vCloud Director Environments

I have been researching what the options for centralized logging in vCloud Director environments. Such deployments consist of large number of systems and to have all their logs in one place is the best practice for their additional processing, archiving and troubleshooting. If the logs are scattered in different locations it is very hard to correlate them and apply consistent policies.

I am going to describe the logging configurability and other considerations for following systems from the VMware vCloud stack:

• vSphere ESXi hosts
• vCenter Server
• vCloud Director (cells)
• vShield (vCloud Network and Security) Edge Gateways
• vShield (vCloud Network and Security) Manager
• vCenter Orchestrator

This article is not about what should be used as the central system that collects those logs and how to mine useful information out of them. Syslog compatible system is the essential requirement here however other enterprise features as high availability, log indexing, load balancing, reporting, archiving and extensibility for non-syslog sources should be considered.

vSphere ESXi Hosts

Log collection options on ESXi hosts are the most advanced out of all considered components. ESX logs are by default stored locally either on local datastore or in memory. They can be redirected to any other datastore however only useful option in large deployments is to redirect them to syslog server. This is done by changing the Syslog.global.logHost variable. Here we can put multiple destination syslog servers. We can also choose the protocol (TCP/UDP/ssl) and port number. The logs are then sent over vmkernel management network.

Note: ESXi produces large number of logs (vmkernel, hostd, vpxa, fdm, …) which are stored in separate files but combined into one syslog stream. Hostd and vpxa logs are set by default to verbose level and are quite chatty, so I usually change them to info level. This can be done by changing Vpx.vpxa.config.log and Config.HostAgent.log.level advanced variables.

vCenter Server

The Windows installable vCenter Server unfortunately cannot be easily configured to send its logs to an external syslog server. Therefore additional agent must be installed. The vCenter Server Appliance (vCSA) edition which runs on linux OS on the other hand utilizes syslog-ng which can be easily reconfigured to forward its logs to remote syslog. William Lam wrote excellent post how to do that here.

vCSA 5.1 when used with the internal PostgreSQL DB is currently supported while managing only up to 4 ESX hosts which might be enough for management cluster. For the vCloud resource clusters either vCSA with external Oracle database or Windows installable vCenter Server must be used.

vCloud Director

vCloud Director cells produce number of logs. There is an audit log which persists in the database for 90 days. It can be forwarded to an external syslog server by changing the $VCLOUD_HOME/etc/response.properties file on each vCloud Director cell. However only one syslog target can be defined with UDP/514 port. Therefore it is recommended to redirect this log to local RHEL syslog (127.0.0.1) and then use its forwarding features. The detail description how to do this is in vCAT 2.0 Public VMware vCloud Implementation Example which can be downloaded from here.

Additional troubleshooting and API access logs are stored locally in $VCLOUD_HOME/logs directory. These logs are produced by Apache log4j logging utility and can be forwarded by creating additional appender in the $VCLOUD_HOME/etc/log4j.properties directory. This is described in the KB article 2004519.

EDIT 1/10/2013: A new KB article has been posted about editing log4j.properties file: KB article 2004564.

vShield Edge Gateways

vShield Edge Gateway is multi-interface vShield Edge virtual device that connects vCloud Director organization VDC networks to external networks. These Edge devices are deployed as needed by vCloud Director through vShield Manager. vCloud Director holds all their configuration details which means that if their properties are edited directly through vShield Manager, vCloud Director is not aware of those changes and could revert them back. Why am I mentioning this? Each Edge device can be configured to send its logs to two remote Syslog servers. Additionally protocol (UDP/TCP) can be specified as well. However vCloud Director only allows to set up two default syslog servers (System > Administration > System Settings > General > Networking). And these will be applied to every Edge device deployed (either Edge Gateways or vApp Edges). So no per tenant or per network configuration is out-of-the-box possible (unless orchestrated with blocking tasks and vCenter Orchestrator).

Multi-interface Edge Gateway allows us to create dedicated logging external network on which the Syslog server (or forwarder to the Syslog server) can be placed. If we would want to collect logs also from vApp Edges the tenant would have to make sure that the vApp Edge can route the syslog traffic through the Edge Gateway to the Syslog server. If it will not (or cannot because it is isolated from Edge Gateway) then the vApp Edge logs are lost.

The provider could decide that he wants to collect only Edge Gateway logs and prohibit any other devices (vApp Edges or any other VM) to send the logs to provider’s Syslog server. This can be accomplished with design shown in the following picture:

Each Edge Gateway has two external networks. One connected to the internet with Sub-allocate IP Pool with external IP addresses which the customer can use for Network Address Translation (NAT) to enable internet access to or from his VMs. The second external interface is connected to the logging external network. Customer cannot create any NAT rules because Sub-allocate IP Pool on that network is not populated. To block routed traffic from VMs behind the Edge Gateway firewall rule “allow only logging external network” is created (allow 10.0.5.0/24).

vShield Manager

vShield Manager (vSM) produces audit and system event logs. These can be forwarded to only one remote syslog server which can be configured in the vSM Settings&Reports > Configuration > General menu. Additionally port can be configured as well, however only UDP protocol is used.

vCenter Orchestrator

Similarly to vCenter Server, vCenter Orchestrator comes in two flavors – Windows installable or virtual appliance. KB article 1010956 describes the location of the files in the Windows installable edition. The same logging redirection can be used as described in the vCenter Server section above.

Note: This articles was written for the following versions: vSphere 5.1, vCloud Director 5.1, vCloud Network And Security 5.1 and vCenter Orchestrator 5.1.