This is part 16 of 20+ blogs I am writing covering the exam prep guide for the VMware Certified Advanced Professional 6 – Network Virtualisation Deployment (3V0-643) VCAP6-NV certification.

At the time of writing there is no VCAP Design exam stream, thus you’re automatically granted the new VMware Certified Implementation Expert 6 – Network Virtualisation (VCIX6-NV) certification by successfully passing the VCAP6-NV Deploy exam.

For previous blogs in this series please refer to the VCAP6-NV Reference Guide I created. This has all the links to VMware NSX content and lists out each exam objective and the associated blog. Check it out here –>Exam Objective Reference Guide.

This blogs covers:

Section 6 – Configure Cross vCenter Networking and Security
Objective 6.1 – Configure Cross vCenter VMware NSX infrastructure components

Configure NSX manager roles (Primary, Secondary, Standalone, Transit)
- Assign Primary role to specified NSX Manager
- Assign Secondary role to specified NSX Manager
Deploy/Configure Universal Controller Cluster
Configure Universal Segment ID Pools
Create/Manage Universal Transport Zones

I will also cover off more than what’s listed above.

Lab Environment Overview

If you read over my first blog post in this series you will know that my lab is running on VMware vCloud Director (vCD) that uses NSX and I have ESXi v6.0 hosts deployed as VMs (unsupported) on vCD as a tenant. These hosts have a single vCenter Server v6.0 and the NSX Manager v6.2 appliance running on them. This first site I will refer to as DC1 (DC= data centre).

I have now built a second site that contains another set of ESXi hosts, vCenter Server and the NSX Manager appliance. This second site I will refer to as DC2.

The vSphere topology looks like this:

Each DC has their own vSphere Platform Service Controller (PSC) and Site (DC1 Site/DC2 Site), there is a single Single Sign-On (SSO) domain spanning the DCs. This design gives a single pane of glass for connected vCenter Servers (Enhanced Link Mode (ELM)).

The two environments have had the following base configurations completed:

ESXi hosts are connected to their respective vCenter Server and joined to Virtual Distributed Switches.
All VMware licences applied.
NSX Managers have been registered to the respective vCenter Server and the Lookup Service configured etc.
Permissions/user accounts etc have been assigned on the NSX Managers.
ESXi hosts, vCenter Server and NSX Manager are all using the same NTP server.
The vCenter Server/s and PSCs have been excluded from the NSX Firewall.
The ESXi hosts have had NSX Host Preparation completed.
No other NSX configuration has been carried out.

Cross-vCenter NSX Deployment Topology

This blog and the two following will cover most of the NSX stuff you can see in the below diagram (Site A and Site B).

Cross-vCenter NSX Topology

We know that an NSX Manager and a vCenter Server are paired together for NSX to function. This is and only is a 1:1 ratio. With cross-vCenter NSX you assign one NSX Manager as the primary NSX Manager with other NSX Managers assigned as secondary NSX Managers. There can be up to seven secondary NSX Managers.

The Primary NSX Manager deploys a Universal Controller Cluster and this creates the NSX Control plane across all NSX Managers involved in the cross-vCenter NSX deployment. Secondary NSX Managers do not have any controller clusters. The Universal Controller Cluster always resides at the site of the primary NSX Manager. The Universal Synchronisation Service (USS) keeps all the secondary NSX Managers in sync with the primary.

There is the concept of universal objects. They are created on the primary NSX Manager and then replicated to the secondaries. You can only modify universal objects on the primary NSX Manager. You can see them on the secondaries, you just can’t edit them. These objects are NSX features such as: universal logical switches, universal logical routers, universal distributed firewall. These NSX features can now be deployed/spread across multiple vCenter Server environments that can be within the data centre (local) or across data centres (remote). Massive.

You can still create local objects on the individual NSX Managers, but as the word ‘local’ implies the objects are only available for that NSX environment and cannot be seen by another NSX Manager.

Note: In a cross-vCenter NSX deployments, all NSX Managers must run the same exact NSX version.

Some New ‘Universal’ NSX Concepts

NSX Manager Roles: Primary|Secondary|Standalone|Transit
Primary, Secondary and Standalone roles are easy to understand. The Transit role is used when a primary or secondary NSX Manager is changed to Standalone and there are remaining universal objects in existence. You might use the transit role when changing which NSX Manager is the primary.

Universal Controller Cluster:
There is only one universal control cluster and it resides with the Primary NSX Manager. This creates the NSX control plane across all secondary NSX Managers.

Universal Transport Zone:
There is only one universal transport zone and it resides with the Primary NSX Manager. For a universal logical switch to span multiple clusters the clusters must be added to the universal transport zone.

Universal Logical Switches:
For a universal logical switch to span multiple clusters the clusters must be added to the universal transport zone. The Segment ID and Universal Segment ID pools must not overlap.

To route between universal logical switches, a universal logical distributed router (UDLR) is required. To route between a universal logical switch and a logical switch an Edge Services Gateway is required.

Universal Logical (Distributed) Routers:
Universal Logical Distributed Routers (UDLR) can only be created on the Primary NSX Manager.

When you deploy a UDLR you have the option to configure/enable local egress and you cannot change this after deployment. This gives the ability to control local egress traffic which it does by restricting routes given to hosts so traffic does not route back to the primary site. To make use of the local egress feature you have to deploy the control VM.

Universal Firewall Rules:
Basically you can create a section in the Distributed Firewall that is marked for universal synchronisation to all secondary NSX Managers. All other rules are local to the NSX Manager. You cannot use Service Composer to create distributed firewall rules.

What Happens When the NSX Manager Role Is Changed?

The NSX Manager role can either be set as primary, secondary, standalone or transit. All NSX Managers at first deployment are standalone.

When an NSX Manager is:

Set as Primary: It starts the universal synchronisation service and syncs to the secondaries.

Set as Standalone from Secondary: The role changes to either standalone or transit.

Set as Standalone from Primary: The role changes to either standalone or transit. It resets the NSX Manager to standalone, stops the universal synchronisation service and unregister all secondary NSX Managers. Watch out!

Disconnect from Primary: Use this when the primary NSX Manager is unrecoverable or you want to point a secondary at a new primary. You run this option from a secondary NSX Manager. Use the force switch to remove the secondary from the primaries database should it come back online.

Determine the Role Assigned to an NSX Manager

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Management.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

If you see no Role column (you can add this column like I have done below) then the NSX Managers have the standalone role.

Note my two NSX Managers have the Standalone role. Also note each is paired to a different vCenter Server.

Assign Primary role to specified NSX Manager

Note: Make sure you know how to add and change all the NSX Manager roles: Primary|Secondary|Standby and Transit.

Some important pre-reqs (not all):

NSX Manager versions must match
NSX Managers must be registered with different vCenter Servers
Segment ID Pool on NSX Managers must not overlap
The NSX Manager to be assigned the Primary role must either have the Standalone or Transit role

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Management.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Make sure you select the correct NSX Manager to be assigned the Primary role.

Click the blue cog (Actions) and select Assign Primary Role.

The Primary role has now been assigned to my NSX Manager (10.0.0.98) that is registered to my vCenter Server (labvc01.lab.local) at DC1.

Assign Secondary role to specified NSX Manager

Note: Make sure you know how to add and change all the NSX Manager roles: Primary|Secondary|Standby and Transit.

Some important pre-reqs (not all):

NSX Manager versions must match
NSX Managers must be registered with different vCenter Servers
Segment ID Pool on NSX Managers must not overlap
One NSX Manager must already have the Primary role assigned
The NSX Manger to be assigned the Secondary role must not have any NSX Controllers deployed
The NSX Manager to be assigned the Primary role must either have the Standalone or Transit role

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Management.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Select the Primary NSX Manager.

Click the blue cog (Actions) and select Add Secondary NSX Manager.

Enter the details to connect to the Secondary NSX Manager i.e. IP, Username, Password.

Check the SSL certificate thumbprint and accept.

The Secondary role has now been assigned to my NSX Manager (10.0.0.96) that is registered to my vCenter Server (labvc02.lab.local) at DC2.

If you are not using Enhanced Link Mode, log into the vCenter via the Web Client that has the secondary NSX Manager to see the role assigned.

Deploy/Configure Universal Controller Cluster

Three (3) Universal Cluster Controllers need to be deployed from the Primary NSX Manager.

You deploy the first controller, wait until the deployment of this controller has completely finished and then you deploy the remaining two controllers (you don’t need to wait for these – just deploy one after the other).

You can either configure an IP Pool prior to the universal cluster controller deployment or during the initial deployment of the first controller by clicking New IP Pool. The IP Pool is used to assign an IP address to each NSX Controller.

I have already created an IP Pool and the process to create one can be seen in Blog 2 under the section ‘Implement and Configure NSX Controllers’. My IP Pool on the primary NSX Manager looks like below:

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Management.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Select the Primary NSX Manager. Make sure it’s the primary!!

Under NSX Controller Nodes click the green sign to Add an NSX Controller.

Add the relevant details for the first NSX Controller to be deployed.

The first NSX Controller starts deploying.

Important: Wait until this NSX Controller has completed before deploying anymore.

Now repeat the process and deploy out NSX Controller 2 and 3.

The below shows the deployed Universal Controller Cluster with 3 nodes.

Note: Check that all ESXi hosts connected to the primary vCenter Server have automatic startup/shutdown rules for the NSX Universal Cluster Controllers.

Configure Primary NSX Manager VXLAN

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Host Preparation.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Select the Primary NSX Manager. Make sure it’s the primary!!

Under VXLAN, click Configure VXLAN.

Enter the relevant details to configure VXLAN on the DC1 cluster. I already have an IP Pool (VTEPpool_DC1) for the VTEP IPs.

This is the IP Pool I have configured.

The hosts are configured for VXLAN. Each host has a VMKernel port configured on the Virtual Distributed Switch that was selected and the IP is assigned from the IP Pool. One host shown below.

You can see VXLAN is now configured for the DC1 cluster.

Configure Primary NSX Manager Local Segment ID

See blog 3 for more information on Segment IDs.

The pool range must be unique to this NSX Manager.

Below I am configuring the local Segment IDs. This is the Virtual Network Identifier (VNI) that is assigned to a Logical Switch.

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Logical Network Preparation.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Select the Primary NSX Manager. Make sure it’s the primary!!

Click Segment ID, followed by Edit.

For DC1 which has the Primary NSX Manager role I am configuring the local Segment ID pool with a range of 5000-7000. Note: you can also see the Universal Segment ID pool. I will configure this later, but it only appears as we have a cross-vCenter deployment.

Configure Secondary NSX Manager VXLAN

This is the same process as the Primary NSX Manager except you do this on the Secondary NSX Manager and obviously you use an IP Pool etc that is configured on that NSX Manager.

This is the VXLAN configuration for the DC2 cluster.

And this is the VTEPpool_DC2 IP Pool in use.

You can see VXLAN is now configured for the DC2 cluster.

Configure Secondary NSX Manager Local Segment ID

Same process as the Primary NSX Manager local Segment ID pool.

Make sure this is done on the Secondary NSX Manager. If you have Enhanced Link Mode you can do this on the Primary, just make sure you have the Secondary NSX Manager selected.

The pool range must be unique to this NSX Manager.

On the Secondary NSX Manager I created a local Segment ID pool with a range of 20000-22000.

Configure Universal Segment ID Pools

The Universal Segment ID pool can only be configured on the Primary NSX Manager.

This pool is used to allocate universal objects i.e. universal logical switches a VNI.

Log into the vSphere Web Client.

Click Networking and Security, then Installation, followed by Logical Network Preparation.

*If you are using Enhanced Link Mode (ELM) you will see multiple NSX Managers.

Select the Primary NSX Manager. Make sure it’s the primary!!

Click Segment ID, followed by Edit.

Enter the range for the Universal Segment ID pool. I have entered the range of 50000-60000.

Note: The Universal Segment ID pool range must be unique and cannot overlap with any other NSX Manager Segment IDs in the cross-vCenter NSX deployment.