Using NSX / Cloudstack / Arista VTEP's

VXLAN encapsulation is IP based and provides a virtual L2 network.  With VXLAN the full Ethernet Frame (not the Frame Check Sequence: FCS) is carried as the payload of a UDP packet.  VXLAN utilizes a 24-bit VXLAN header and provides for up to 16 million virtual L2 networks.

Frame encapsulation is done by an entity known as a VXLAN Tunnel Endpoint (VTEP.)  A VTEP has two logical interfaces: an uplink and a downlink.  The uplink is responsible for receiving VXLAN frames and acts as a tunnel endpoint with an IP address used for routing VXLAN encapsulated frames.  These IP addresses are infrastructure addresses and are separate from the tenant IP addressing for the nodes using the VXLAN fabric.

The following implementation requires Cloudstack 4.3.0, NSX 4.1.2 and EOS 4.14. Make sure that all Hypervisors and NSX nodes have the VXLAN connector configured.

At Schuberg Philis we have dozens of NSX gateways to build Hybrid environments. The use of VTEP's is more flexible and provides more flexibility.

Arista switch configuration

The switch needs to be able to connect to all the Hypervisors for the VXLAN tunnel and the the NSX cluster and service nodes for management and control.


NSX configuration: The VTEP gateway

In NSX Manager, add a new gateway. Click the Network Components tab, then the Transport Layer category. Under Transport Node, click Add, then select Manually Enter All Fields. The Create Gateway wizard appears.

In the Create Gateway dialog, select Gateway for the Transport Node Type, then click Next.
In the Display Name field, give the gateway a name, then click Next.
Enable the VTEP service. Select the VTEP Enabled checkbox, then click Next.
Copy the certificate from the switch with the following show command:

Copy the certificate and paste it into the Security Certificate text field. Copy only the bottom portion, including the BEGIN and END lines and paste it in the NSX manager:

Click Next.
In the Connectors dialog, click Add Connector to add a transport connector. This defines the tunnel endpoint that terminates the VXLAN tunnel and connects NSX to the physical gateway. You must choose a tunnel Transport Type of VXLAN. Choose an existing transport zone for the connector.
Define the connector’s IP address (that is, the underlay loopback IP address on the switch for tunnel termination).
Click OK to save the connector, then click Save to save the gateway.

Verify the management connection in NSX (openflow should be down):

Verify the management connection on the switch from the CLI and the BASH:


NSX configuration: The transport layer

When you finished the NSX integration, you need to configure the transport layer. For each host-facing switch port that is to be associated with a VXLAN instance, define a Gateway Service for the port.
In the NSX Manager, add a new gateway service. Click the Network Components tab, then the Services category. Under Gateway Service, click Add. The Create Gateway Service wizard appears.
In the Create Gateway Service dialog, select VTEP L2 Gateway Service as the Gateway Service Type.

Give the service a Display Name to represent the VTEP in NSX.
In the Transport Node field, choose the name of the gateway you created earlier.
In the Port ID field, choose the physical port on the gateway (in this example eth10) that will connect to a logical L2 segment and carry data traffic.

Click OK to save this gateway in the service, then click Save to save the gateway service.

NSX configuration: The logical layer

To complete the integration with NSX, you need to configure the logical layer, which requires defining a logical switch (the VXLAN instance) and all the logical ports needed. This can be a 'special' bridged network or a network created with CloudStack.
In this example I have used a network that was originally created with CloudStack and adjusted the for the VTEP use case.

To define the logical switch, do the following:
In the NSX Manager, add a new logical switch. Click the Network Components tab, then the Logical Layer category. Under Logical Switch, click Add. The Create Logical Switch wizard appears.
In the Display Name field, enter a name for the logical switch, then click Next.

Under Replication Mode, select Service Nodes, then click Next.
Specify the transport zone bindings for the logical switch. Click Add Binding.

Select the VxLan transport type, the correct transport zone and assign a VNI (0 when the interface is a trunk).
In our case also a STT binding for the Hypervisors is needed
Click OK to save this gateway in the service, then click Save to save the gateway service.

NSX configuration: Connect ports to the logical layer

The final step, define the logical switch ports. They can be virtual machine VIF interfaces from a registered OVS, or a VTEP gateway service instance on this switch, as defined above in the Configuring the Transport Layer. A VLAN binding can be defined for each VTEP gateway service associated with the particular logical switch.

In the previously created logical switch you can create ports. Under Logical Switch Port, click Add. The Create Logical Switch Port wizard appears.

In the Logical Switch UUID list, select the logical switch you created above.

In the Display Name field, give the port a name that indicates it is the port that connects the gateway, then click Next.
In the Attachment Type list, select VTEP L2 Gateway.
In the VTEP L2 Gateway Service UUID list, choose the name of the gateway service you created earlier.
In the VLAN list, you can optionally choose a VLAN if you wish to connect only traffic on a specific VLAN of the physical network. Leave it blank to handle all traffic.
Click Save to save the logical switch port. Connectivity is established. Repeat this procedure for each logical switch port you want to define.

Arista switch troubleshooting

Verify the VTEP gateway connection with the NSX cluster controllers (handshake for example).

Verify the VNI relation that was configured in NSX under logical switch binding:

Verify the VxLan connection to a Hypervisor

VxLan Broadcast configuration (via the NSX service nodes):

VxLan controller connection:


Daniel Clarke
Good write-up, I'm still getting my head around how this compares to the Microsoft NVGRE solution.

Not Published

0/1000 characters
Go Top