Nicira Network Virtualization Platform (NVP)

Yesterday, I had a short technical discussion with a Singapore-based Nicira staff member. They have a niche solution that is doing exactly what the Cisco Nexus and L2 MPLS (i.e. Ethernet over MPLS or Any Transport over MPLS) are trying to achieve: multi-tenancy Data Center Interconnect (DCI) by creating multiple layer 2 virtual networks (or pseudo-wire) across an IP network. Layer 2 networks are essential for many Data Center applications, especially Cloud Virtualization. Imagine performing server clustering, VMWare vMotion, Hyper-V Live Migration on multiple physical sites, which otherwise break by IP subnets. It should also be seriously considered as part of IT Disaster Recovery plans. As for multi-tenancy, you may have multiple network customers or tenants sharing the same underlying physical infrastructure. Each tenant should only see its respective overlay network without visibility into other virtual network - a similar concept to host virtualization and cloud computing.

How does Nicira NVP work? From what I understand in high level perspectives, a data path STT (Stateless Transport Tunneling) tunnel is established between 2 or more Open vSwitches (OVS) across an IP network. This MAC-over-IP tunnel is used to encapsulate all MAC layer traffic and transport them over an IP network, which effectively connect 2 servers (whether virtual or physical) on different sites as if they were on the same subnet or VLAN. 

As of now, the OVS can be integrated into ESX, KVM and XenServer hypervisors. There is also near-future plan for Hyper-V support (not sure if the plan would be cancelled, as Nicira is now acquired by its arch-rival VMWare). Alternatively, an ISO-based image can also be run off as a virtual or physical server as an OVS gateway connecting legacy systems to the Nicira network virtualization platform (NVP).

As for OVS management and control, the NVP Controller Cluster (housed on server clustering) is used to centrally managing and controlling all OVS along the control paths. I was told that even if there were a disconnection on the control paths, the OVS would continue to operate (even though not modifiable at this stage).

From what I observed, the new Cisco Overlay Transport Virtualization (OTV) is probably Nicira's arch-rival at this point. True enough that traditional networking MPLS and L2 pseudo-wires can be employed to perform the same tricks, they are either limited by performance, lack of MPLS aware devices or simply staff knowledge in the IP networks. Both Cisco OTV and Nicira NVP, on the other hand, can be easily established across any traditional IP-based networks.

And prices also do matter. I was told that one would need at least USD150K for a small POC setup to "try out" Nicira NVP. For the same price tag, I could also have purchased at least half-dozen of Cisco ASR1000 hardware (OTV is now supported on ASR platform from version XE3.5S onward). A princely sum that is pretty hard-to-justify for a software purchase to the management I suppose.

Ethernet over MPLS (EoMPLS) Cisco Configuration Makes Simple

Ethernet over MPLS (EoMPLS) is part of Cisco's Any Transport over MPLS to provide L2 connectivity (pseudo-wire) over MPLS cloud. If you wish to extend EoMPLS on L3 VLAN interface (SVI-based EoMPLS or SwEoMPLS), you must have an OSM or an Enhanced FlexWAN module on the MPLS core-facing interface. Otherwise, you can configure PFC-based EoMPLS on a physical interface or sub-interface. In this example, we would use PFC-based EoMPLS on Cisco IOS 15.x, as I believe most of us won't have any special interface cards. Consider this network diagram below. We'll extend L2 connectivity on VLAN 10 to connect both servers over the MPLS cloud.

Here, we assume that basic MPLS configuration has been put in place. Configuring EoMPLS would be pretty straightforward.

On both PE1 and PE2 routers:
!

interface GigabitEthernet0/1.10
encapsulation dot1Q 10
xconnect 10.1.1.x 10 encapsulation mpls
no shut

!
Replace above 'x' with the peer PE router ID i.e. on PE1 x = 2 and on PE2 x = 1. The router ID is determined by the "mpls ldp router-id" command on the router.

On both CE1 and CE2 switches, let's assume Gi0/1 switch interface is used to connect to their respective PE routers.
!
Vlan 10
  name EoMPLS
!
interface GigabitEthernet0/1
switchport trunk encapsulation dot1Q
switchport mode trunk
!
interface GigabitEthernet0/2
description Host port
switchport mode access
switchport access vlan 10
!

To verify the EoMPLS connectivity, enter "show mpls l2transport vc" on both PE routers. The status should indicate UP. You can also perform "show spanning tree vlan 10" on both CE switches to ensure the sanity of spanning tree i.e. only one of the switches should be the root. And finally, both hosts should be able to ping each other on same IP subnet. For further details, refer to this Cisco article.

Multicast for MPLS VPN Extranet

In my earlier blog post, I've mentioned briefly about Multicast VPN across MPLS VPN. In this setup, the VRF must share the same default Multicast Distribution Tree (MDT), 239.1.1.1 in this example, for the multicast VPN to form dynamically. Consider the following setup:

The source server (on PE1) is streaming on multicast mode. The VRF-Green, on the other side of PE2, would be able to join to the same multicast distribution tree (239.1.1.1) and recieve the multicast stream. For this Intranet Multicast VPN to work, the VRF-Green on both PE routers should share the basic common configuration as follows:

Router PE1 and PE2
  ip vrf VRF-Green
  mdt default 239.1.1.1
  route-target export 65001:100
  route-target import 65001:100
!
ip multicast-routing
ip multicast-routing vrf VRF-Green
ip pim rp-address 1.1.1.1
ip pim vrf VRF-Green rp-address 10.1.1.1
!
router bgp 55
  address-family ipv4 mdt
    neighbor x.x.x.x activate
    neighbor x.x.x.x send-community extended
!
  address-family vpnv4
     neighbor x.x.x.x activate
     neighbor x.x.x.x send-community extended
!

However, the VRF-Blue (on PE2) won't be able to join the same MDT, as Cisco routers do not support multiple duplicate default MDT across VRFs. However, there is a workaround for this Extranet setup. Do note that data flow in MPLS VPN happens in label-exchange. Create another VRF-Green on the source PE1 and import the multicast traffic over to VRF-Blue on PE2. There are 2 options of importing, either import it on the source PE or on the reciever PE. I observed that the latter reciever PE option is more reliable if there are more than 1 reciever PE routers.

Configuring the Receiver MVRF on the Source PE (Option 1)
On PE1, add "route-target import 65001:100" on VRF-Blue, enable multicast routing "ip multicast-routing vrf VRF-Blue", and specify the same Rendezvous Point (RP) as VRF-Green for VRF-Blue using "ip pim vrf VRF-Blue rp-address 10.1.1.1".

For this setup, assign a new default MDT group for VRF-Blue on both PE routers (239.2.2.2 in this example). Import the label on VRF-Green over (route-target import 65001:100) as shown in above diagram. In this way, two multicast distribution trees (MDTs) can be created from the same source server. When you perform a "show ip mroute vrf VRF-Green" on PE1, you'll see Extranet receivers in VRF-Blue.

Configuring the Source MVRF on the Receiver PE (Option 2)
Another option is to do the import on the reciever PE (PE 2 in this example). As mentioned earlier, I observed that the stream is slightly more stable if there is more than 1 reciever PE. You'll need to do the import on every reciever PE.

Troubleshooting
Ensure RPF and multicast routing is sound for GRT level on all PE and P routers, as well as VRF level for all PE. In this example (some outputs truncated):
GRT level
#sh ip mroute
(10.10.10.1, 239.1.1.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Port-channel1, RPF nbr 10.10.0.1
  Outgoing interface list:
    Port-channel2, Forward/Sparse, 00:54:29/00:03:15

VRF level
#sh ip mroute vrf VRF-Green
(192.168.1.1, 239.23.25.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Tunnel1, RPF nbr 192.168.0.1
  Outgoing interface list:
    VLAN123, Forward/Sparse, 00:54:29/00:03:15

For further details, refer to the Cisco Multicast VPN Extranet support.

Multipath with Redundant Routers
If you have redundant paths with some load splitting or sharing, you may have to consider enabling multipath options using following steps on every router:

1. enable
2. configure terminal
3. ip multicast multipath s-g-hash next-hop-based
4. ip multicast vrf vrf-Name multipath s-g-hash next-hop-based
5. end
6. show ip rpf source-address group-address
7. show ip route ip-address

Reference: Load Splitting IP Multicast Traffic over ECMP

Load Sharing in MPLS VPN with Route Reflector

No matter what I did according to this Cisco article, I simply can't do load-sharing among the PE routers within our enterprise MPLS VPN. The central Route Reflector (RR) simply refuses to send more than 1 identical route, even though I have multiple PE routers attached to that route.

Subsequently, I come across this excellent MPLS VPN blog. It explains that there is no way to change the route reflector behavior.  The workaround is to simply assign different Route Distinguisher (RD) values for the same VRF on different routers. To enable route propagation, ensure that you export and import common Route Target (RT) across the PE routes. Finally, enable "maximum-paths eibgp " on each vpnv4 address family under the router bgp process, so as to inject more than one bgp route into the routing table.

And it works magic!

Example configuration according to above diagram

On Router PE A

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE B

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE C

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE D

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

Multicast for MPLS VPN

We need to transport multicast traffic across our enterprise MPLS core network for users to receive their favorite Channel News Asia and CNN live broadcasts on their workstations. I found this updated Cisco resource - Configuring Multicast VPN (MVPN) - for IOS 12.4, which is different from the older IOS versions. The newer version support MDT SAFI, which will be automatically created when the default MDT (Multicast Distribution Tree) exists. The default MDT defines the path used by PE routers to send multicast data and control messages to every other PE router in the multicast domain. The setting up of MDT involves the following steps:

1. Configuring a Default MDT Group for a VRF (required)


2. Configuring the MDT Address Family in BGP for Multicast VPN (required)


3. Configuring the Data Multicast Group (optional)

Data MDTs, which are dynamically created, are intended for high-bandwidth sources such as full-motion video inside the VPN to ensure optimal traffic forwarding in the MPLS VPN core. When the multicast transmission exceeds the defined threshold, the sending PE router creates the data MDT and sends a UDP message, which contains information about the data MDT to all routers on the default MDT. The threshold at which the data MDT is created can be configured on a per-router or a per-VRF basis. However, Data MDTs are created only for shared tree (S, G) multicast route entries within the VRF multicast routing table. They are not created for source tree (*, G) entries regardless of the value of the individual source data rate.

MPLS MTU for Enterprise MPLS VPN

We are in the midst of building a new 10GE MPLS VPN for our enterprise campus core network. After we setup our first MPLS VPN, we are able to connect up 2 different VRFs (a.k.a virtual networks) belonging to 2 different departments. Ping test shows that the RTT was close to 0 msec. However, Active Directory traffic was slowed to a claw.

We investigated and found no fault at the routing and mpls configuration. Later, I recalled an experience of slow access over GRE tunnel relating to MTU sizing. Further search on Cisco.com revealed this:

When configuring the network to use MPLS, set the core-facing interface MTU values greater than the edge-facing interface MTU values, using one of the following methods:

• Set the interface MTU values on the core-facing interfaces to a higher value than the interface MTU values on the customer-facing interfaces to accommodate any packet labels, such as MPLS labels, that an interface might encounter. Make sure that the interface MTUs on the remote end interfaces have the same interface MTU values. The interface MTU values on both ends of the link must match.
• Set the interface MTU values on the customer-facing interfaces to a lower value than the interface MTU on the core-facing interfaces to accommodate any packet labels, such as MPLS labels, than an interface might encounter. When you set the interface MTU on the edge interfaces, ensure that the interface MTUs on the remote end interfaces have the same values. The interface MTU values on both ends of the link must match.

We adopted the former approach and set MTU size of all core facing interfaces to 1520 bytes and leave all customer facing interfaces to default 1500 bytes.