Layer 2 VPN to the Cloud – Part II

Almost 3 years ago I have published an article how to set up layer 2 VPN between on-prem vSphere environment and vCloud Director Org VDC.

As both vCloud Director and NSX evolved quite a bit since to simplify the whole set up, here comes the part II.

Let me first summarize the use case:

The tenant has an application that resides on 3 different VLAN based networks running in its own (vSphere) datacenter. The networks are routed with existing physical router. The tenant wants to extend 2 of these networks to cloud for cloud bursting or DR purposes, but not the 3rd one (for example because there runs a physical database server).

The following diagram shows the setup.

The main advancements are:

  • vCloud Director natively supports NSX L2 VPN (VCD 8.20 or newer needed).
  • NSX now (since 6.2) supports configuration of unstretched networks directly (no static routes are necessary anymore)
  • This means the full setup can be done by the tenant in self-service fashion

Here are the steps:

  • The tenant will deploy freely available NSX Standalone Edge in its datacenter connected to trunk port with 2 VLANs mapped (10 and 11). Additional network configuration is necessary (forged transmits and promiscuous mode or sink port creation – see the link)
  • In the cloud Org VDC tenant deploys two routed Org VDC networks with identical subnets and gateways as networks A and B. These networks must be connected to the Org VDC Edge GW via subinterface (there can be up to 200 such networks on single Edge). The Org VDC Edge must have advanced networking enabled.
  • Tenant enables and configures L2VPN server on its Org VDC Edge GW. Note that this is a premium feature that the service provider must enable in Organization first (see this blog post).
  • Before the L2VPN tunnel is established the following must be taken into account:
    • The Org VDC Edge GW IP addresses are identical with the on-prem existing physical router. Therefore Egress Optimization Gateway addresses must be entered in the Peer Site configuration. That will prevent the Org VDC Edge GW from sending ARP replies over the tunnel.
    • The same must be performed on the Standalone NSX Edge via CLI (see egress-optimize command here).
    • The non-stretched network (subnet C) must be configured on the Org VDC Edge GW so it knows that the subnet is reachable through the tunnel and not via its upstream interface(s). This option however is not in vCloud UI, instead vCloud networking API must be used.
      Edit 3/26/2018: This does not works for standalone NSX Edges. See the end of the article for more details.
      Alternatively the provider could configure non-stretched network directly in the NSX UI:
    • Finally, the tunnel can be established by configuring L2VPN server details on the on-prem Standalone NSX Edge L2VPN client (endpoint IP, port, credentials, encryption) and providing VLAN to tunnel mappings.
    • Note to find the Org VDC network subinterface tunnel mapping vCloud API must be used again:

Edit 3/26/2018:

After multiple questions regarding unstretched networks and some testing I need to make some clarifications.

The routing of unstretched networks through the tunnel is achieved via static routes configured on the Edge GW. So in principle it still works the same way as described in the original article, the difference doing it via UI/API is that the setting of the IPs and routes is automatic.

The server Edge routing table looks like this:

show ip route

S [1/0] via
C [0/0] via
C [0/0] via
C [0/0] via
C [0/0] via
C [0/0] via
S [1/0] via

show ip address


17: vNic_4094@br-sub: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:50:56:88:31:21 brd ff:ff:ff:ff:ff:ff
inet brd scope global vNic_4094
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe88:3121/64 scope link
valid_lft forever preferred_lft forever

You can see that the IP address was autoassigned to the 4096 tunnel interface and static route was set to route the unstretched network to the other side via IP The assignment of the .194 address on the other Edge will happen only if that Edge is managed by NSX and is actually performing routing! This is in fact not true for the use case above (with standalone Edge and existing physical router). Therefore the following manual approach must be taken:

  1. Create Org VDC transit network with arbitrary small subnet (e.g. in the cloud. Assign IP .1 as the gateway on the Org VDC Edge. This network will not be used for workloads, it is used just for routing to unstretched network.
  2. Create corresponding VLAN transit network on-prem. Assign IP .2 as its gateway interface on the existing router (note the IP addresses of the routing intefaces in #1 and #2 are different).
  3. Create L2 VPN tunnel as before, however also stretch the transit network but do not optimize its GW (no need as on-prem and cloud are using different IPs).
  4. Create static routes on the Org VDC Edge GW to route to on-prem unstretched networks  via the transit network router IP.

Note that this is approach very similar to the original blog post. The only difference is that we must create separate transit network as vCloud Director does not support multiple subnets on the same Edge GW interface.

6 thoughts on “Layer 2 VPN to the Cloud – Part II

  1. Hi Tom, we working on VCD (9) nsx (6.4) with l2vpn multi-tunnel, it can not pingable cross tunnel if egress-optimize enabled, opposite it work. May I know it is normal?
    We also tested unstretched netowrks, before your post we tested dedicated tunnel for osfp/bgp, and following your post test unstretched network seem not work. would share more experience on egress-optimize and unstretched networks, thanks.

  2. Hi Thomas,

    are the entries for unstreched networkscreally necessary for routibg other networks through the tunnel?
    In other description I found this is only neccessary for untaged networks. We conected two vclouds without entrys in unstreched networks and everything worked fine (also routing of other networks than the streched one through the tunnel).
    Or is this only neccessary if we use the standalone gateway?

    Regards stefan

  3. Hi Tom,

    I have a similar setup, just no vcloud. When i follow your post, i have reachability between stretched workloads on cloud to non-stretched workloads on-prem just fine to and fro. This works well when i create gateway for the stretch subnet on L2vpn server ESG. However when i have default gateway on different ESG and transit network on L2vpn server ESG, i can reach from stretch workload in cloud to non-stretch workloaf on-prem but not the other way around. Is this a limitation that the gateways should always be on L2vpn server ESG and cannot be on physical switches. P.S. – i have standalone client on-prem and default gateway on local router just like the example you took.

  4. Hi Tom ,

    Thanks for your great Article, I need to do the same setup for a DR site (cold)but i have an additional requirement to connect with a direct link(through ISP) between the customer data center and customer cloud tenant (vDC) not connect over the internet (their will be VMs replication through the same link), so what is the best approach for link connectivity and in this case am i still require to use NSX edge ?
    one more question please , what about the end user access if we supposed that protected applications are only accessible only from local network?

    1. I assume the direct link is not Layer 2, so you will still have to establish L2 VPN between the standalone Edge on-prem and Org VDC Edge in the cloud. That should be possible by just routing the VPN over the private link.
      End user access would be over the L2 VPN. Note that L2 VPN is not a solution you want to keep permanently. It should be used only temporarily for the migration. Once that is done, you should remove the L2 VPN and route the internal users over routed connection via the direct link to the Org VDC Edge.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.