Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.