vCloud Director IPv6 Support

With depletion of public IPv4 addresses service providers are starting to consider offering IPv6 addresses for their tenants workloads. Let me describe what are the options related to IPv6 support for service providers that use vCloud Director.

In the new vCloud Architecture Toolkit (vCAT) Document for Service Providers I have proposed a design how to provide IPv6 services to tenants. So let me summarize the constraints:

  • Currently in vCloud Director the tenants cannot assign IPv6 subnets to Org VDC or vApp networks
  • In consequence this means that the tenants cannot use vCloud Director IP Address Management (IPAM) to assign IPv6 addresses to their VMs. However, IPv6 addresses can still be assigned from within the guest operating system.
  • vCloud Director deployed Edge Gateways do not support IPv6. It means internal or external interfaces of the Edges need to have IPv4 addresses.
  • vCloud Director relies for network services on vCloud Networking and Security (vCNS) or NSX components. vCNS does not support IPv6 however NSX does. vCNS will soon go out of support anyway.

The proposed design that works around the above limitations is following. Let me paste the picture from the linked vCAT document:

Provider managed tenant Edge GW

The provider deploys NSX Edge Service Gateway outside of vCloud Director (directly from NSX GUI/API) and connects it to a VXLAN or VLAN based network which is then imported to vCloud Director as an external network. Both the Edge Gateway and the external networks are dedicated to a particular tenant and managed by the provider.

The tenant can attach his workloads to an OrgVDC network which is directly connected to the external network. As this tenant NSX Edge is managed externally outside of vCloud Director scope it can offer full set of services NSX provides – and among them are IPv6 services.

There is one undocumented cool feature that I recently discovered which enables even more IPv6 functionality to the tenant.

There is in fact the possibility for service provider to assign IPv6 subnet to the external network and thus the tenant can use vCloud Director IPAM in a limited way. He can manually assign IPv6 address (IP Mode Static – Manual) to a VM network interface from vCloud Director UI/API and let vCloud Director to configure the VM networking through guest customization. vCloud DIrector even makes sure the IP address is unique.

Note: IP Mode Static – IP Pool is not supported as it is not possible to define IPv6 IP pool.

Here is how to configure IPv6 subnet on external network:

  1. Create vCloud DIrector external network (with IPv4 subnet)
  2. Find vCloud UUID of the external network. For example use the following API call: GET /api/admin/extension/externalNetworkReferences
  3. Insert into vCloud Director database gateway, prefix length, nameservers and dns suffix information. You must create new entries in config table with the following values:

    cat = network
    name = ipv6.<ext network UUID>.gateway | subnetprefixlength | nameserver1 | nameserver2 | dnssuffix
    value = <value of the network property>

    The following example is valid for MS SQL database:

    external network UUID: 85f22674-7419-4e44-b48d-9210723a8e64
    subnet: fd6a:32b6:ab90::/64
    gateway IPv6 address: fd6a:32b6:ab90::1
    DNS 1: fd13:5905:f858:e502::208
    DNS 2: fd13:5905:f858:e502::209
    dns suffix: acme.fojta.com


    INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.dnssuffix', 'acme.fojta.com', 0);
    INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.nameserver1', 'fd13:5905:f858:e502::208', 0);
    INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.nameserver2', 'fd13:5905:f858:e502::209', 0);
    INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.subnetprefixlength', '64', 0);
    INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.gateway', 'fd6a:32b6:ab90::1', 0);
  4. In the tenant Org VDC create Org VDC network directly connected to the external network.
  5. The tenant can now connect VMs to the Org VDC network and assign IPv6 addresses directly from UI (or API).
    Deploy template with IPv6

    Deploy template with IPv6

    VMs with IPv6 Addresses

    VMs with IPv6 Addresses

Note that when using this provider managed Edge Gateway concept, the external network is dedicated to a particular tenant. For scalability reasons it is recommended to use VXLAN based external networks created directly in NSX. vCloud Director supports maximum of 750 external networks.

The tenant cannot directly manage Edge Gateway services and must rely on the provider to configure them.

vCloud Director 5.6 with NSX – Edge Redeploy Gotcha

Just a short post about an issue with combination of vCloud Director 5.6 and NSX.

In case you are upgrading vCNS to NSX and are using vCloud Director 5.6 you are still affected by the Edge Redeploy bug described in the following two KBs:

The legacy vCNS Edges (version 5.5) are during vCloud Director redeploy operations incorrectly upgraded to NSX Edges (version 6). This breaks compatibility with vCloud Director.

The fix involves adding following entry into vCloud Director database:

Oracle:

INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'vcloud', 'networking.edge_version_for_vsm6.1', '5.5', 0);
commit;

Microsoft SQL Server:

INSERT INTO config (cat, name, value, sortorder) VALUES ('vcloud', 'networking.edge_version_for_vsm6.1', '5.5', 0);

For NSX 6.1 use …vsm6.1, For NSX 6.0 or 6.2 use …vsm6.0 or …vsm6.2 in the string above.

The bug is fixed in builds of vCloud Director 5.5.3 and later and 8.0 and later.

Federation of Multiple vCloud Director Instances


While vCloud Director offers impressive scalability with possibility to manage up to 20 vCenter Servers there are valid reasons why service providers deploy multiple vCloud Director instances due to fault domains (availability zones) or latency (multiple geographical regions) requirements.Federation

In such case a single tenant (end-customer) has multiple organization accounts in multiple vCloud Director instance each with access to subset of VDCs, catalogs, VMs, etc. The service provider would usually hide this under one custom overarching portal however when the tenant wants to discover and manage his resources in programmable fashion he has to access multiple vCloud API endpoints of the vCloud Director instances.

vCloud Director version 8.0 offers a new feature: Organization Associations. If you search through vCloud API v 9.0 you will see new Admin Types:  OrgAssociationType  and OrgAssociationsType and related new Admin Element OrgAssociationMember. So what it is and how does it work?

The service provider can associate multiple Organizations (belonging to the same customer) together and simplify discovery of all cloud resources through single API endpoint.

Example

In my lab I have two different vCloud Director instances with URLs vcloud.fojta.com and vcloud2.fojta.com. In the first instance I have created two organizations belonging to the same customer: ACME and ACME2. In the second instance I have created organization ACME. All three organizations use the same identity source (in my case LDAP) and the same user exists in all of them with Organization Administrator credentials.

  1. The system administrator creates the organization associations. In this example I have associated org: ACME in the second instance with ACME and ACME2 in the first instance. This is done with the following vCloud API call:
    PUT https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations
    Content-Type: application/vnd.vmware.admin.organizationAssociations+xml
    
    Body:
    <?xml version="1.0" encoding="UTF-8"?>
    <OrgAssociations xmlns="http://www.vmware.com/vcloud/v1.5">
        <OrgAssociationMember href="https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations/02b433db-0b37-4304-b07b-0717255ec297" type="application/vnd.vmware.admin.organizationAssociation+xml">
            <MemberUrl>https://vcloud.fojta.com/api/org/02b433db-0b37-4304-b07b-0717255ec297</MemberUrl>
            <MemberName>ACME</MemberName>
            <MemberEndpointCertificate>-----BEGIN CERTIFICATE-----
    MIIFfzCCBGegAwIBAgITTgAAARuwZOW3iRv9KQABAAABGzANBgkqhkiG9w0BAQsF
    ADBCMRMwEQYKCZImiZPyLGQBGRYDY29tMRUwEwYKCZImiZPyLGQBGRYFZm9qdGEx
    FDASBgNVBAMTC2ZvanRhLURDLUNBMB4XDTE1MDExNTE2NTAwNFoXDTE3MDExNDE2
    NTAwNFowZzELMAkGA1UEBhMCQ1oxDzANBgNVBAcTBlByYWd1ZTESMBAGA1UEChMJ
    Zm9qdGEuY29tMRgwFgYDVQQLEw92Q2xvdWQgRGlyZWN0b3IxGTAXBgNVBAMTEHZj
    bG91ZC5mb2p0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1
    UeaPIsHjRQb8PkybTv7tfCe6oyq8UUxc7tiwX+nWvHWKJ9X6ASis1v/gT0CCa4cG
    fP+tezXayXMrwFSRu6OvanBSTVYvaAbUQl5CsVnQaaeCC5bTAMfGlDsl/q+LnqzW
    i0eG4hTpWG78v88AZkaHjIZdr5CQuDaPGJUqzOgrHjpYTLDJs+oK+S7ScpMyKhia
    hgeRKDfrEeQGGvSEMdHhg3Bg8RK8eyLQLjwCSCVkhYTrM5wyt4fow65beoMnOBbx
    LO+QpB6/amy5mJuVLVx+WJivVvuId2hmELorm2iJMjUAabybLmbMPmqHTTGyZaYw
    vxaDRDr0DbTMUYFyOh6LAgMBAAGjggJHMIICQzAdBgNVHQ4EFgQUi1Dhxpbkz9Dh
    tYOljP+MW/9GF+AwHwYDVR0jBBgwFoAUs0GCJG1KfknG9couJQXq4CZq4SQwgfoG
    A1UdHwSB8jCB7zCB7KCB6aCB5oaBr2xkYXA6Ly8vQ049Zm9qdGEtREMtQ0EoMSks
    Q049REMyLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2
    aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNvbT9jZXJ0aWZpY2F0
    ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u
    UG9pbnSGMmh0dHA6Ly9EQzIuZm9qdGEuY29tL0NlcnRFbnJvbGwvZm9qdGEtREMt
    Q0EoMSkuY3JsMIG7BggrBgEFBQcBAQSBrjCBqzCBqAYIKwYBBQUHMAKGgZtsZGFw
    Oi8vL0NOPWZvanRhLURDLUNBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2
    aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNv
    bT9jQUNlcnRpZmljYXRlP2Jhc2U/b2JqZWN0Q2xhc3M9Y2VydGlmaWNhdGlvbkF1
    dGhvcml0eTAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA4GA1Ud
    DwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOC
    AQEArNmceh3GkHx5Qf9s/NUlG7lh+i5nPUzlbnlhTJQeOJXSKG3DzU8ocb1xWguT
    x1ICyLZTQq11q3D3/3xgi9KJJaWzo8X5O/Mj81x4V8Jp9d8OgERc7lrVyrAJPbJA
    k7q/4tY41VOu8P5i+A21Pxuo3xELkOt5rcb2qt6QH1QizSA8Dzjq8uwCpdDo8eCP
    ZWUwc2lNUOyCmhFD7boNecHRJZN92i3W0YKfV+BC0cIXnqU2Y+4YEKAHWwH/gRm0
    ZI41oyatyoHpTjCGFtKzrSo/mxitIoj5ZTY9wwSNPlcmziw29hOTM1fOx//rqgrW
    17CREB/BoFj12Hd9bVXFgMGUSg==
    -----END CERTIFICATE-----</MemberEndpointCertificate>
        </OrgAssociationMember>
        <OrgAssociationMember href="https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations/13e52807-3d0a-4c0f-abdb-62d8fccb36ea" type="application/vnd.vmware.admin.organizationAssociation+xml">
            <MemberUrl>https://vcloud.fojta.com/api/org/13e52807-3d0a-4c0f-abdb-62d8fccb36ea</MemberUrl>
            <MemberName>ACME2</MemberName>
            <MemberEndpointCertificate>-----BEGIN CERTIFICATE-----
    MIIFfzCCBGegAwIBAgITTgAAARuwZOW3iRv9KQABAAABGzANBgkqhkiG9w0BAQsF
    ADBCMRMwEQYKCZImiZPyLGQBGRYDY29tMRUwEwYKCZImiZPyLGQBGRYFZm9qdGEx
    FDASBgNVBAMTC2ZvanRhLURDLUNBMB4XDTE1MDExNTE2NTAwNFoXDTE3MDExNDE2
    NTAwNFowZzELMAkGA1UEBhMCQ1oxDzANBgNVBAcTBlByYWd1ZTESMBAGA1UEChMJ
    Zm9qdGEuY29tMRgwFgYDVQQLEw92Q2xvdWQgRGlyZWN0b3IxGTAXBgNVBAMTEHZj
    bG91ZC5mb2p0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1
    UeaPIsHjRQb8PkybTv7tfCe6oyq8UUxc7tiwX+nWvHWKJ9X6ASis1v/gT0CCa4cG
    fP+tezXayXMrwFSRu6OvanBSTVYvaAbUQl5CsVnQaaeCC5bTAMfGlDsl/q+LnqzW
    i0eG4hTpWG78v88AZkaHjIZdr5CQuDaPGJUqzOgrHjpYTLDJs+oK+S7ScpMyKhia
    hgeRKDfrEeQGGvSEMdHhg3Bg8RK8eyLQLjwCSCVkhYTrM5wyt4fow65beoMnOBbx
    LO+QpB6/amy5mJuVLVx+WJivVvuId2hmELorm2iJMjUAabybLmbMPmqHTTGyZaYw
    vxaDRDr0DbTMUYFyOh6LAgMBAAGjggJHMIICQzAdBgNVHQ4EFgQUi1Dhxpbkz9Dh
    tYOljP+MW/9GF+AwHwYDVR0jBBgwFoAUs0GCJG1KfknG9couJQXq4CZq4SQwgfoG
    A1UdHwSB8jCB7zCB7KCB6aCB5oaBr2xkYXA6Ly8vQ049Zm9qdGEtREMtQ0EoMSks
    Q049REMyLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2
    aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNvbT9jZXJ0aWZpY2F0
    ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u
    UG9pbnSGMmh0dHA6Ly9EQzIuZm9qdGEuY29tL0NlcnRFbnJvbGwvZm9qdGEtREMt
    Q0EoMSkuY3JsMIG7BggrBgEFBQcBAQSBrjCBqzCBqAYIKwYBBQUHMAKGgZtsZGFw
    Oi8vL0NOPWZvanRhLURDLUNBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2
    aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNv
    bT9jQUNlcnRpZmljYXRlP2Jhc2U/b2JqZWN0Q2xhc3M9Y2VydGlmaWNhdGlvbkF1
    dGhvcml0eTAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA4GA1Ud
    DwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOC
    AQEArNmceh3GkHx5Qf9s/NUlG7lh+i5nPUzlbnlhTJQeOJXSKG3DzU8ocb1xWguT
    x1ICyLZTQq11q3D3/3xgi9KJJaWzo8X5O/Mj81x4V8Jp9d8OgERc7lrVyrAJPbJA
    k7q/4tY41VOu8P5i+A21Pxuo3xELkOt5rcb2qt6QH1QizSA8Dzjq8uwCpdDo8eCP
    ZWUwc2lNUOyCmhFD7boNecHRJZN92i3W0YKfV+BC0cIXnqU2Y+4YEKAHWwH/gRm0
    ZI41oyatyoHpTjCGFtKzrSo/mxitIoj5ZTY9wwSNPlcmziw29hOTM1fOx//rqgrW
    17CREB/BoFj12Hd9bVXFgMGUSg==
    -----END CERTIFICATE-----</MemberEndpointCertificate>
        </OrgAssociationMember>
    </OrgAssociations>
    

    As can be seen I am supplying URI for each Organization, its name and endpoint certificate.

  2. I can review organization associations with GET /admin/org/{id}/associations call or I can add or remove single association with similar POST / DELETE calls.
  3. Now when the end-user authenticates against the organization that has these associations he can run query calls that will run against all associated organizations. To make the query federated he must add federated=global string to the Accept header.
    GET https://vcloud2.fojta.com/api/query?type=organization
    Accept: application/*;version=9.0;federated=global
    x-vcloud-authorization: 4a1f241b371b46f5a36abac85f5962c7
    

    The reply lists all three organizations:

    <?xml version="1.0" encoding="UTF-8"?>
    <QueryResultRecords xmlns="http://www.vmware.com/vcloud/v1.5" name="organization" page="1" pageSize="128" total="3" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=records" type="application/vnd.vmware.vcloud.query.records+xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcloud2.fojta.com/api/v1.5/schema/master.xsd">
        <Link rel="alternate" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=references" type="application/vnd.vmware.vcloud.query.references+xml"/>
        <Link rel="alternate" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=idrecords" type="application/vnd.vmware.vcloud.query.idrecords+xml"/>
        <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="ACME Corporation" isEnabled="true" isReadOnly="false" name="ACME" numberOfCatalogs="1" numberOfDisks="0" numberOfGroups="7" numberOfVApps="1" numberOfVdcs="2" storedVMQuota="0" href="https://vcloud.fojta.com/api/org/02b433db-0b37-4304-b07b-0717255ec297" numberOfRunningVMs="1"/>
        <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="ACME Inc." isEnabled="true" isReadOnly="false" name="ACME2" numberOfCatalogs="0" numberOfDisks="0" numberOfGroups="1" numberOfVApps="1" numberOfVdcs="1" storedVMQuota="0" href="https://vcloud.fojta.com/api/org/13e52807-3d0a-4c0f-abdb-62d8fccb36ea" numberOfRunningVMs="0"/>
        <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="Acme Inc." isEnabled="true" isReadOnly="false" name="ACME" numberOfCatalogs="0" numberOfDisks="0" numberOfGroups="1" numberOfVApps="0" numberOfVdcs="0" storedVMQuota="0" href="https://vcloud2.fojta.com/api/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea" numberOfRunningVMs="0"/>
    </QueryResultRecords>
    

Design Considerations

  • Only GET requests for the top level Org Level data (org, sessions, tasks, templates/catalogs) and all Query Service requests by Org users can be federated.
  • All System Admin query request span single vCloud Director instance and cannot be federated.
  • All non-GET requests cannot be federated.
  • All associated organizations must use the same identity source as the user is authenticated against all organizations and vCloud Director stores the session tokens for subsequent requests. Local accounts are not supported.
  • OAuth identity source offers the lowest administration overhead  when managing multiple organizations belonging to the same end-user as the user/role management can be performed centrally (see vCAT-SP for more information).
  • Association works only in one direction: when organization A is associated with organization B it does not mean that organization B is associated with A. Bidirectional association must be explicitly configured for both organization.

vCloud Architecture Toolkit for Service Providers

One of the reasons I have been quiet on my blog lately was today released for public: vCloud Architecture Toolkit for Service Providers.

If you are designing vCloud Director based public cloud I hope you will find design considerations and recommendation included helpful. Let me highlight a few in my opinion interesting topics from the Architecting a VMware vCloud Director Solution document:

  • Virtual Machine metric Cassandra database
  • RabbitMQ messaging infrastructure
  • Considerations around NSX Edge Cluster and NSX Controllers
  • Consumption of NSX services
  • VSAN impact on vCloud Director catalogs
  • OAuth authentication

Layer 2 VPN to the Cloud

When VMware NSX 6.0 came out about more than one year ago, one of the new great features it had on top of the its predecessor VMware vCloud Network and Security (vCNS) was L2VPN service on Edge Service Gateway which allows stretching layer 2 network segments between distant sites in different management domains. NSX 6.1 further enhanced the functionality by introducing Standalone Edge which can be deployed on top of vSphere without an NSX license and acts as L2VPN client.

Many vCloud Service Providers are now deploying their public cloud with vCloud Director and NSX instead of vCNS so I am often asked how could they leverage NSX in order to provide L2VPN service to their tenants.

As of today neither vCloud Director 5.6 nor 8.0 (in beta at the moment) can deploy NSX Edges and manage the L2VPN service. However it is still possible for the SP to provide L2VPN as a managed service for his tenants.

Let me demonstrate how would it work on the following artificial example.

The customer has an application that resides on 3 different VLAN based networks (subnet A, B and C) routed with existing physical router. He would like to extend subnets A and B into the cloud and deploy VMs there. The VMs in the cloud should access the internet in order to connect to external SaaS services through the provider connection (egress optimization) but should still be able to reach database running on subnet C which is hosted on premises.

The diagram below shows the whole architecture (click for larger version):

L2VPN to the Cloud

On the left is the customer on premises datacenter with physical router and three VLAN based networks. On the right is the public cloud with NSX design I proposed in one of my previous blog articles. While the unimportant parts are grayed out what is important is how the customer Org VDC and the NSX Edge Gateways is deployed.

  • The provider deploys tenant dedicated NSX Edge Service Gateway outside of vCloud Director manually and configures it based on customer requirements. The provider creates two logical switches (VXLAN 5005 and 5006) which will be used for extending customer subnets A and B. The switches are trunked to the NSX Edge and the Edge interface IP addresses are assigned identical to the IP addresses of the physical router on-premises (a.a.a.1 and b.b.b.1).
  • The two logical switches are configured in vCloud Director as External Networks with the proper subnets A and B and pool of unused IPs.
  • Two Org VDC networks are created inside tenant’s Org VDC as directly connected to the two External Networks.
  • L2VPN server is configured on the NSX Edge (with encryption algorithm, secret, certificate and stretched interfaces). Also Egress Optimization Gateway Address is configured (both physical gateway IPs are entered – a.a.a.1 and b.b.b.1). This will filter ARP replies of the two gateways sharing the same IPs in the tunnel and allow NSX Edge to act as the gateway to the internet.
  • The tenant will install Standalone Edge which is distributed as OVF inside his datacenter and set it up: he must map VLANs to tunnel IDs supplied by the provider, configure Edge Server public IP and port and encryption details.

Now what about the subnet C? How can VMs deployed in the cloud get to it if the physical router is unreachable due to the enabled egress optimization? Following trick is used:

  • Another subnet z.z.z.0/30 is used for P2P connection between the NSX Edge in the cloud and the physical router.
  • IP address z.z.z.1/30 is configured on the physical router on one of the stretched subnets (e.g. A).
  • The second IP z.z.z.2/30 is configured on the NSX Edge on the same subnet.
  • Finally static route is created on the NSX Edge pointing subnet C to the next hop address z.z.z.1.

Some additional considerations:

  • In case the tenant has licensed NSX on-premises he can obviously use ‘full’ NSX Edge Service Gateway. The advantages are that it is much easier to deploy and configure. It can also stretch VXLAN based networks as opposed to only VLANs which are supported by Standalone Edge.
  • Standalone Edge can be connected either to Standard Switch or Distributed Switch. When Standard Switch is used promiscuous mode and forged transmits must be enabled on the trunk port group. VLANs ID 4095 (all) must be configured to pass multiple VLANs.
  • When Distributed Switch is used it is recommended to use Sink Port instead of promiscuous mode. Sink Port receives traffic with MAC addresses unknown to vDS.
  • Sink Port creation is described here. It requires vDS edit via vCenter Managed Object UI. While Sink Port can be also created with net-dvs –EnableSink command directly on the ESXi host with Standalone Edge VM running it is not recommended as the host configuration can be overridden by vCenter Server.
  • RC4-MD4 encryption cipher should not be used as it is insecure and has been deprecated in NSX 6.2.