Rate Limiting of External Networks in vCloud Director and Nexus 1000V

There is a new feature in vCloud Director 5.1 which was requested a lot by service providers – configurable limits on routed external networks (for example Internet) for each tenant. Limits can be set both for incoming and outgoing directions by vCloud Administrator on tenant’s Edge Gateway.

Edge Rate Limit Configuration

However this feature only works with VMware vSphere distributed switch – it does not work with Cisco Nexus 1000V or VMware standard switch. Why? Although the feature is provided by the Edge Gateway, what is actually happening in the background is that vShield Manager instructs vCenter to create a traffic shaping policy on the distributed vswitch port used by the Edge VM.

vSphere Distributed Switch Traffic Shaping

Standard switch does not allow port specific traffic shaping and Nexus 1000V management plane (Virtual Supervisor Module) is not accessible by the vShield Manager/vCenter. The rate limit could be applied on the port of the Cisco switch manually, however any Edge redeploy operation, which is accessible by the tenant via GUI would deploy a new Edge and use different port on the virtual switch and tenant could thus easily disable the limit.

For the standard switch backed external network vCloud Director GUI will not even present the option to set the rate limit, however for the Nexus backed external network the operation will fail with similar error:

Cannot update edge gateway “ACME_GW”
java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10086): Traffic shaping policy can be set only for a Vnic connected to a vmware distributed virtual portgroup configured with static port binding. Invalid portgroup ‘dvportgroup-9781′.

Nexus 1000V Error

Btw the rate limit can be set on the Edge (when not using vCloud Director) also via vShield Manager or its API – it is called Traffic Shaping Policy and configurable in the vSM > Edge > Configure > Interfaces > Actions menu.

vShield Manager Traffic Shaping

Do not forget to consider this when designing vCloud Director environments and choosing the virtual switch technology.

Load Balancing vCloud Director Cells with vShield Edge

Big deployments of vCloud Director should have at least two vCloud Director cells for high availability and load balancing reasons. This implies usage of a load balancer. One can choose either physical box (for example F5) or virtual one (Citrix Netscaler, Riverbed Stingray, Zenloadbalancer, …). With the new release of VMware vCloud Networking and Security (vCNS) which is the successor of VMware vShield it is possible to use the Edge (version 5.1) as a load balancer.

Compared to the old vShield Edge (5.0) there are quite a few enhancements. Besides being able to load balance not only HTTP connection as was the case in the previous versions, load balancing of HTTPS and generic TCP connections is also supported. Additionally the new Edge can have up to 10 network interfaces, can connect to VXLAN networks, provide traffic shaping, relay DNS, create SSL VPN and can scale up to 3 sizes (compact, large, x-large) with statefull active passive high availability.

I am going to describe how to use Edge as a load balancer in front of two vCloud Director cells. The following picture shows my lab network setup.

This is based on quite standard architecture where the vCloud Director cells sit in DMZ zone usually separated by two firewalls from the internet and the management zone. In order to deploy the Edge, vCNS Manager (former vShield Manager) must be deployed first. If two different vCenters are used for management of resource group cluster and management cluster, also two different vCNS Managers must be used as there is 1:1 relationship between the vCenter and vCNS Manager.

Deployment Process

1. Deploy vCNS Manager (OVF virtual appliance), configure and register with management cluster vCenter

2. Either using vSphere Client (use the .NET version as there is no vShield plugin for Web Client available yet) or directly through vCNS Manager web GUI go to Host and Clusters view, select Datacenter and click Network Virtualization tab. Click + icon to add a new Edge.

3. Configure the Edge deployement size, HA, network interfaces (portgroups, IPs and subnets), default firewall policy and placement. In my lab I have used compact size, no HA and two interfaces (INT and EXT as shown in the picture).

4. Once the Edge is deployed (Manager deploys OVF and then with VIX API pushes configurations to the Edge VM), select it and click the gear icon Actions to go to Manage menu.

Before we configure the load balancer we must add additional IP(s) to the external interface. This is vCloud Director requirement as both portal/API and VMware Remote Console (VMRC)  Proxy use the same port 443. I have used the default Edge external IP address for the vCloud Director portal and added a second one for VMRC Proxy. This can be done in Configure tab, interfaces menu.

5. Now we can configure the load balancer. Firstly Pools of real servers must be created and then Virtual Server can be configured.

I have created two pools: VCD_80-443 with two services enabled: HTTP and HTTPS, both using LEAST_CONN Balancing Method on Ports 80 and 443. I have enabled HTTP health check with the default settings on URI /cloud/server_status. The members were the VCD cells with IPs 10.0.1.60 and 10.0.1.62 and respective ports 80 and 443 on each IP.

The second pool:  VMRC_443 has a TCP service with LEAST_CONN Balancing Method and default TCP health check on port 443. The VCD cell IPs 10.0.1.61 and 10.0.1.62 with ports 443 were added.

6. Two Virtual Servers were then created. One for each external IP from step 4. “vcloud” Virtual Server uses VCD_80-443 Pool with 10.0.2.80 external IP address. “VMRC” Virtual Server uses VMRC_443 Pool with 10.0.2.81 external IP address.

7. The configurations must be uploaded to the Edge by clicking the Publish Changes button.

Happy load balancing.

My Thoughts About VMware vShield Data Security

VMware this summer released a new addition to their vShield family of security products – vShield App with Data Security. Its purpose is to discover sensitive data in the VMs and thus enabling of the assessment of organization’s compliance with regulations such are PCI-DSS (Payment Card Industry Data Security Standards). In other words vShield Data Security is DLP (Data Loss Protection) solution which scans files in the virtual machines for a known patterns (credit card numbers, social security numbers, etc.). It is very similar to an antivirus solution.

I am posting some of my findings from the testing of vShield Data Security to help others to evaluate it.

• Although Data Security is bundled with vShield App and is called vShield App with Data Security technically it does not use vShield App infrastructure at all and is very similar to vShield Endpoint antivirus solution. It also relies on EPSEC framework where files are passed from VM via loadable kernel module to Endpoint security appliance which does the actual scanning. Data Security relies on RSA licensed code to do the scanning.
• The installation must be done in the following steps:
• vShield Manager appliance must be installed. This is a common management component for the whole vShield product family and is delivered as OVA virtual appliance. There must be one vShield Manager per vCenter. It has 1 CPU and 3MB RAM and should be HA or better FT protected. It can be managed via web UI and also integrates with vCenter. The current version that supports Data Security is VMware-vShield-Manager-5.0.0-473791.ova.
• Each host running virtual machines that will be scanned has to be prepared by installing the EPSEC kernel module. This does not require host restart nor maintenance mode. It is done from vSphere client by clicking on the host, then selecting the vShield tab and installing the vShield Endpoint service. It must be done manually for each host.
• vShield Data Security endpoint appliance is deployed next. It is done similarly to the previous step again from the host vShiled tab by installing vShield Data Security service. It is a 1 CPU 512 MB RAM virtual machine. There must be one service VM per each participating host. The service VM does not migrate to other hosts and can be installed on local storage. It has two NICs, one for communication with vShield Manager (it needs IP management address) and one for standard virtual switch used for communication with vmkernel which is created automatically. This switch is also used for vShield App service machines or other Endpoint (antivirus) machines if they exist on the hosg.
• All the guest VMs that will be scanned by Data Security endpoints need a vShield driver. This vShield driver is included in VMware Tools released in September ESXi5.0 update (Patch Release ESXi500-201109001). I am scratching my head here as why is VMware not releasing the driver as a separate download. vShield Data Security is compatible with ESX(i) 4.1 and there has been no VMware Tools update with vShield driver released as of now. As a VMware employee I have access to internal VMware Tools builds and was able to download the most recent build that contained the driver. The other way is to download vSphere ESXi5, install it (for example in VMware Workstation), upload the patch or better just the VMware Tools VIB (unziped from the patch) to accessible datastore and from CLI install it (~ # esxcli software vib install -v /vmfs/volumes/Local_Datastore/VMware_locker_tools-light_5.0.0-0.3.474610.vib). The upgraded VMware Tools can then be found in one of the other local volumes. Following command copies the tools to Local Datastore from which it can be retrieved with vSphere Client (replace the x)

  ***edit 11/10/2011 The vib can be openned and extracted in 7zip. So just rename the *.vib to *.7 and then extract the windows.ISO file.

  ~ # cp /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/packages/5.0.0/vmtools/windows.iso /vmfs/volumes/Local_Datastore/

• Now global scanning policies can be set with vSphere client by clicking Datacenter, vShield tab, Data security, Policy. They consist of what patterns to look for categorized by Regulations (e.g. EU Debit Card Numbers), which datacenters, clusters or resource pools to exclude, and which files to scan (list of extensions). The policy must be published after the editing is finished.
• Now the scanning can be started from the same page.
• The scanning process runs only once, sequentially one VM per host. There is no way to see the progress other then to go to Task & Events > Events of each VM and to look for “vShield Data Security scan started / ended on the VirtualMachine” message.
•  If a file with sensitive information is found no action is taken other than it is included in Violation Counts report (number of violating files per regulation) and Violating Files  report, which contains exact file location and can be downloaded as csv.
• Although scanning is reported as “In Progress”, once all the machines are scanned nothing is scanned anymore until a new VM is added or the policy is changed and published. There is no way to schedule the scans other than using REST API scripts.
• Things I would like to see to improve in future releases:
• enterprise features such are easier bulk deployment of the kernel module and endpoint appliance, centralized logging, scheduler, reporting tool
• more granular selection of VMs to be scanned
• definition of actions based on scanning results
• more visibility into scanning process
• on demand scanning of accessed files
• vmdk offline scanning of turned off VMs

vShield Troubles

I have recently installed VMware vCloud Director on ESXi server in my home lab. However this is not what I want to blog about today. Yesterday I rebooted the ESXi server and all the vCloud Director related changes were gone – the Oracle DB VM, vCloud Director, vShield Manager and vShield appliance were not registered in the inventory. That would not be so bad as that the virtual network did not work at all. I could not ping any virtual machine. And because I run vCenter virtual I was in very bad situation.

I knew that my problem was because of vShield related changes vCloud Director did to the networking. The vShiled internal switch was gone. So I though I had two options. Either to get vShield working again from command line or to reinstall ESXi. I tried both however was not successful on any of them. I was surprised that newly reinstalled ESXi with the old VMs did not fix the networking. And then I noticed this:

All the virtual NIC of all the VMs were disconnected. So I checked the connected checkbox but the setting would not stick.

Then I checked the .vmx files of all the VMs and noticed vShield related changes to the networking.

ethernet0.filter0.name = “vshield-dvfilter-module”
ethernet0.filter0.param1 = “uuid=52393e32-ee4f-4420-808d-dd2683015301.000″

I removed the two entries as seen in the next screenshot rebooted the VM and finaly the network was working again.