PVLAN-like Behavior with ACL on VXLAN Network

Private VLANs (PVLANs) are useful in multitenant environments where there is a need to provide access to single server for all customers while not allowing them to see each other and NAT or dedicated networks are not an option. Agent based backup with central backup server or VM monitoring are example of such use-cases.

There might be a constraint that does not allow the usage PVLAN. For example hardware that does not support it (Cisco UCS) or VXLAN logical network. The latter I used in one of my designs. There was a need for single backup network stretched over pods without L2 connectivity. VXLAN can overlay L3 fabric and thus create single network spanning the pods however VXLAN does not support PVLANs. If VMware NSX is used then distributed (in-kernel) firewall can be used instead. If vCloud Network and Security is providing the VXLAN networks then there is App Firewall (vShield App) which unfortunately provides significant performance and throughput hit as it is inspecting every packet/frame in user-space service VM. It also adds complexity to the solution.

Access Control Lists

As of vSphere 5.5 vSphere distributed switch (vDS) supports access control lists (ACL) at the portgroup level. The ACL configuration is available only via the vSphere Web Client in the vDS portgroup > Manage > Settings > Policies > Edit > Traffic filtering and marking section. The following configuration can be used to provide the similar behavior to private VLAN.

Rule 1: Allow VMs to Server

Action: Allow
Type: MAC
Protocol/Traffic type: any
VLAN ID: any
Source Address: any
Destination Address: MAC address of the promiscous server or router

Rule 2: Allow Server to VMs

Action: Allow
Type: MAC
Protocol/Traffic type: any
VLAN ID: any
Source Address: MAC address of the promiscous server or router
Destination Address: any

Rule 3: Allow ARP

Action: Allow
Type: MAC
Protocol/Traffic type: any
VLAN ID: any
Source Address: any
Destination Address: FF:FF:FF:FF:FF:FF

Rule 4: Drop all

Action: Drop
Type: MAC
Protocol/Traffic type: any
VLAN ID: any
Source Address: any
Destination Address: any

Screenshot of the final configuration:

ACL rules

Note: As with PVLANs there is still a security issue of the tenant misconfiguring VMs IP address and causing Denial-of-Service for another VM with the same IP address. There are ways to remediate it but out of scope of this article.


Troubleshooting VXLAN vmknic in vSphere 5.5

As of vSphere 5.5 VXLAN has its own vmkernel networking stack therefore ping connectivity testing between two different vmknics on the transport VLAN must be done from ESXi console with the following syntax:

vmkping ++netstack=vxlan <vmknic IP> -d -s <packet size>


esxcli network diag ping --netstack=vxlan --host <vmknic IP> --df --size=<packet size>

Thanks go to Ray Budavari for the hint.

vCloud Director: Isolated Org VDC Network and DHCP

Today I learned one little trick. You know when you create an isolated Org VDC network in vCloud Director there is always one armed Edge deployed in it which does nothing but provides a DHCP service? If you do not need it then you have to go to Configure Services on the Network and disable DHCP and it will destroy the Edge. Well it is actually possible to get rid of it right away if you define static IP Pool that covers all available IPs on the subnet while creating the network.

Example: If I define gateway address and define static pool no Edge will be deployed.

I could think of two use cases when it could be useful:

1. It speeds up the deployment and put less strain on the vSphere environment, which can be handy during automated provisioning of large number of isolated networks.

2. If you create very large subnet (A class for example) the Edge deployment will error out with message “VSM response error (10014): Configuration failed on vShield Edge vm vm-11689. (invalid dhcp config: ) - Failed to configure services on the network.“. This is caused by very large DHCP range. So the workaround is to use the trick with large IP Pool.

Edge DHCP Error

Monitor Wear and Tear of Your VSAN SSDs

As I started experimenting with VMware Virtual SAN (VSAN) in my lab and am using consumer grade SSDs which are not on VSAN (beta) HCL I am worried about wear and tear of the memory cells who have limited write endurance.

I wondered if it is possible to access the SMART attributes of the disks and quick search showed that there is a KB article 2040405 written which still applies although not specific to vSphere 5.5.

From the ESXi console run esxcli storage core device list to get list of storage devices and then run esxcli storage core device smart get -d device to get the SMART data.

This is output of my Intel 520 drive.

Intel 520 SMART Attributes


Unfortunately another host with OCZ SSD does not display any data with an error:

Error getting Smart Parameters: CANNOT open device


vCAC 6 – How To Generate Signed Certificates

This is the procedure I used to generate and import signed certificates for vCloud Automation Center 6.0.

Identity Appliance

  • Generate private key and certificate signing request with OpenSSL. Common name is FQDN of the Identity Appliance.

openssl.exe req -newkey rsa:2048 -keyout sso.key -nodes -days 3650 -out sso.csr -sha256

Loading ‘screen’ into random state – done
Generating a 2048 bit RSA private key
writing new private key to ‘sso.key’
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
Country Name (2 letter code) [AU]:CZ
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:Prague
Organization Name (eg, company) [Internet Widgits Pty Ltd]:fojta.com
Organizational Unit Name (eg, section) []:vCAC Identity Appliance
Common Name (e.g. server FQDN or YOUR name) []:vcacsso.fojta.com
Email Address []:

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

  • Sign the certificate signing request sso.csr with your CA. Download the signed certificate in Base 64 encoded format (sso.cer).
  • In the SSO > SSL section of Identity Appliance VAMI interface (https://<Identity Appliance FQDN>:5480) choose action: Import PEM encoded Certificate.
  • Paste the private key sso.key to the RSA Private Key field
  • Paste the signed certificate sso.cer to the Certificate Chain section. Append CA root certificate as well.
  • Click Replace Certificate.

Identity Appliance SSL Screen

vCAC Appliance

The process is identical – the only difference is the certificate Common Name and that we are using vCAC Appliance VAMI interface (http://<vCAC Appliance FQDN>:5480 for the import.

IaaS Components

In distributed architecture there can be multiple IaaS components: load balanced website components with Model Manager, Manager service with DEM Orchestrator (active/passive) and multiple Agents and DEM Workers. All those components are Windows based with identical procedure to create domain certificate.

  • Open Microsoft Management Console (mmc.exe) and add Certificates Snap-In (manage Computer account, Local computer).
  • Browse to the Personal Certificates folder and select action Request New Certificate.
  • Request Active Directory Enrollment Policy > Web Server. In the Subject tab configure certificate properties (FullDN, Common Name, Country, etc.), in the General tab type friendly name and in the Private Key tab make private key exportable.
  • Finish by clicking Enroll. Your Domain based CA should now issue the signed certificate.

See my older post that describes this in more detail with screenshots.