Layer 2 VPN to the Cloud

When VMware NSX 6.0 came out about more than one year ago, one of the new great features it had on top of the its predecessor VMware vCloud Network and Security (vCNS) was L2VPN service on Edge Service Gateway which allows stretching layer 2 network segments between distant sites in different management domains. NSX 6.1 further enhanced the functionality by introducing Standalone Edge which can be deployed on top of vSphere without an NSX license and acts as L2VPN client.

Many vCloud Service Providers are now deploying their public cloud with vCloud Director and NSX instead of vCNS so I am often asked how could they leverage NSX in order to provide L2VPN service to their tenants.

As of today neither vCloud Director 5.6 nor 8.0 (in beta at the moment) can deploy NSX Edges and manage the L2VPN service. However it is still possible for the SP to provide L2VPN as a managed service for his tenants.

Let me demonstrate how would it work on the following artificial example.

The customer has an application that resides on 3 different VLAN based networks (subnet A, B and C) routed with existing physical router. He would like to extend subnets A and B into the cloud and deploy VMs there. The VMs in the cloud should access the internet in order to connect to external SaaS services through the provider connection (egress optimization) but should still be able to reach database running on subnet C which is hosted on premises.

The diagram below shows the whole architecture (click for larger version):

L2VPN to the Cloud

On the left is the customer on premises datacenter with physical router and three VLAN based networks. On the right is the public cloud with NSX design I proposed in one of my previous blog articles. While the unimportant parts are grayed out what is important is how the customer Org VDC and the NSX Edge Gateways is deployed.

  • The provider deploys tenant dedicated NSX Edge Service Gateway outside of vCloud Director manually and configures it based on customer requirements. The provider creates two logical switches (VXLAN 5005 and 5006) which will be used for extending customer subnets A and B. The switches are trunked to the NSX Edge and the Edge interface IP addresses are assigned identical to the IP addresses of the physical router on-premises (a.a.a.1 and b.b.b.1).
  • The two logical switches are configured in vCloud Director as External Networks with the proper subnets A and B and pool of unused IPs.
  • Two Org VDC networks are created inside tenant’s Org VDC as directly connected to the two External Networks.
  • L2VPN server is configured on the NSX Edge (with encryption algorithm, secret, certificate and stretched interfaces). Also Egress Optimization Gateway Address is configured (both physical gateway IPs are entered – a.a.a.1 and b.b.b.1). This will filter ARP replies of the two gateways sharing the same IPs in the tunnel and allow NSX Edge to act as the gateway to the internet.
  • The tenant will install Standalone Edge which is distributed as OVF inside his datacenter and set it up: he must map VLANs to tunnel IDs supplied by the provider, configure Edge Server public IP and port and encryption details.

Now what about the subnet C? How can VMs deployed in the cloud get to it if the physical router is unreachable due to the enabled egress optimization? Following trick is used:

  • Another subnet z.z.z.0/30 is used for P2P connection between the NSX Edge in the cloud and the physical router.
  • IP address z.z.z.1/30 is configured on the physical router on one of the stretched subnets (e.g. A).
  • The second IP z.z.z.2/30 is configured on the NSX Edge on the same subnet.
  • Finally static route is created on the NSX Edge pointing subnet C to the next hop address z.z.z.1.

Some additional considerations:

  • In case the tenant has licensed NSX on-premises he can obviously use ‘full’ NSX Edge Service Gateway. The advantages are that it is much easier to deploy and configure. It can also stretch VXLAN based networks as opposed to only VLANs which are supported by Standalone Edge.
  • Standalone Edge can be connected either to Standard Switch or Distributed Switch. When Standard Switch is used promiscuous mode and forged transmits must be enabled on the trunk port group. VLANs ID 4095 (all) must be configured to pass multiple VLANs.
  • When Distributed Switch is used it is recommended to use Sink Port instead of promiscuous mode. Sink Port receives traffic with MAC addresses unknown to vDS.
  • Sink Port creation is described here. It requires vDS edit via vCenter Managed Object UI. While Sink Port can be also created with net-dvs –EnableSink command directly on the ESXi host with Standalone Edge VM running it is not recommended as the host configuration can be overridden by vCenter Server.
  • RC4-MD4 encryption cipher should not be used as it is insecure and has been deprecated in NSX 6.2.

5 thoughts on “Layer 2 VPN to the Cloud

  1. Quick clarification – Does the L2VPN Server/Client (ESGs) have to be the DG for the segments you wish to extend with L2VPN? If I have Physical router as the DG for the VLAN backed PG, do I need to move that over to the Edge?


  2. This blog is a few years old, I hope Tom can still support questions, here goes….

    We have a L2 stretch network from on prem to cloud and looking to unstretch so we don’t get the additional cost feature to stretch. I know OVH provides a free option to stretch. The question is, is it better to have a stretched network when possible or not? What’s a cloud best practice in this situation?

  3. Hi Tomas,

    Need your expert help.

    I followed your article and Customer Onboarding with VMware NSX® L2VPN Service for VMware Cloud Providers by Harold Simon to setup the connectivity via NSX L2 VPN server hosted on vCloud and NSX Standalone Client hosted on on-premise DC.

    On client side, we have some strange configuration. Since the source DC is small, they rented a server to install vCenter and then installed Standalone client but they did not configure any standard / distributed switch and all the switching activities are handled on the physical switch side. So no promiscuous or sink port config done.

    As per the article, we stretched the on-premise VLANs to vxLAN on vCloud and defined egress optimized IP addresses for the GW for each VLAN on both L2 VPN server and Standalone client side.

    Edge Gateway on vcloud side automatically has the GW IPs assigned as part of the routed / stretched vxLANs.

    However, due to some reason, we some how cannot reach the VLAN gateway hosted in source DC from target DC VM over L2 VPN link.

    Do you think with this information, you can gauge some conclusion and identify the possible bottleneck or you would like to have some more information.

    Thanks in advance.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.