Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.

Part 3: Configuring RIP routing on Windows Server 2008

This is following up on part 2 that described the concept of network high availability for single  non-clustered Windows server using dynamic IP routing. This final part is to outline implementation steps. On the Cisco routers, ensure that auto-summarization of RIP routing is disabled, as we are using /32 host addresses on the server loopback adapters.

Step 1: Install Windows loopback adapter

Add microsoft loopback adaptor using hardware wizard (hdwwiz.exe) with elevated admin rights. Assign a host IP address on it using /32 or 255.255.255.255 subnet mask. Add a host record statically on the DNS server using this IP address.

Step 2: Install “Routing and Remote Access Services” Server Role 

Open up Server Manager and add a server role named "Routing and Remote Access". Check on the following items:

Step 3: Add new routing protocol “RIP”

On the Server Manager console, right click on “Routing & Remote Access”, choose “Enable and Configure Routing ….” Click “Next”, select “Custom Configuration”. Select “LAN routing” and complete the wizard. Under “Routing and Remote Access > IPv4”, right click on “General” and choose “New Routing Protocol”. Select “RIPv2 for Internet Protocol”.

Step 4: Enable RIP on Network Adapter

Click on RIP and add new adapters (only public interfaces and loopback that you want users to access – never put in your management interface!). By default, Cisco routers would perform RIPv2 routing on multicast protocol, so select this option on the General tab.

For route optimization, it may be a good idea that your network routers to simply inject default routes into your server routing table. Click on "Advanced" tab and enable to following options:

You would also want to allow the server to advertise the host address (remember the loopback interface in step 1), which can be statically map to a host record on the DNS server.

Step 5: Securing RIP routing (Optional) (Important for better stability & security)

To prevent your server from listening to "rogue" RIP router/server that give bad routes, specify your trusted RIP neighbors on the Security tab (Right-click on RIP and choose Properties).

Furthermore, you probably won’t want your servers to become routers and carry network traffics unknowingly. You can further secure it by limiting it to advertise its own addresses only. For each interface added to RIP routing, choose “Properties”. Permit only the NIC address and the loopback address for outgoing routes.

Step 6: Verify IP routing

There are two ways to verify the IP routing table on your server, which should be populated by the network routers. You can view the IP routing table through the Routing And Remote Access console or through a command prompt. In the Routing And Remote Access console, expand the IP Routing node, right-click the Static Routes node, and then click Show IP Routing.

On the client network, do a continuous ping check (ping -t x.x.x.x) on the loopback address of the server. To simulate a NIC failure, disable one NIC and you may just see a couple of timeouts before resuming.

Part 2: Network Redundancy for Windows Server: Dynamic IP Routing

Besides NIC teaming in earlier part 1, another way to achieve network redundancy for Windows Server is to employ dynamic IP routing. Windows Servers support both static and dynamic IP routing. In dynamic routing, new routes are learned dynamically from the route peers (typically routers and L3 switches). Any changes in network topology (including failure of server NICs or switches) would trigger a routing update for all participating routing nodes. In other words, if there were any failure in any switches or NICs, traffic would have avoided the broken path and continue to route on unbroken links.

Examples of dynamic routing include Open Shortest Path First (OSPF) and Routing Information Protocol (RIP). Windows Server 2003 supports both OSPF and RIP while Windows Server 2008 only supports RIP. Besides achieving network redundancy, you need not configure routing statically, which is otherwise prone to network topology changes and configuration errors. Consider the following example:

Dual-homed ServerX is connected to 2 different switches on different subnets. RIPv2 routing is enabled on both NICs. By default, ServerX would dynamically register both DNS records with AD-integrated DNS servers. With DNS round-robin, network load-balancing can even be achieved among multiple clients. However, in the event of either NIC or switch failure, half of the clients would lose connection with ServerX, which is usually not desirable. Consider the next example:

MS Loopback adapter is installed. RIPv2 routing is enabled on two physical NICs, as well as the loopback adapter. Dynamic DNS update is also disabled. Single static A record is created with loopback IP address.  As dynamic routing supports equal-cost multi-path load-balancing, multiple clients reach ServerX loopback via both NICs. If either NIC or switch fails, dynamic routing would have to re-converge. After a short converging time, clients can regain access to ServerX on its loopback address via the sole remaining NIC.

In conclusion, with both dynamic routing and static DNS record on loopback adapter, both network load-balancing and network redundancy can be achieved. In my next post, I would summarize the steps to achieve this in part 3.

Part 1: Network Redundancy for Windows Servers: NIC Teaming

To provide network redundancy to a single Windows server, you would first need to have 2 or more NICs. To protect against a single switch failure, you would further need to ensure different NICs are connected to different network switches.

Generally, the most common way is to use hardware vendors' specific NIC teaming solutions, where multiple NICs are grouped and teamed into a single virtual network adapter. And you manage the Virtual Adapter just like how you manage a physical adapter. The most common NIC teaming solutions include:

1) For Intel NIC: Advanced Networking Services (ANS)

2) For Broadcom NIC: Broadcom Advanced Control Suite 3 (BACS3)

For each of the vendor solutions, there are various teaming modes. Some protect against single NIC failure, single switch failure or even both (including load-balancing features). For Intel ANS, it supports the following teaming modes:

1) Adapter Fault Tolerance (AFT)

A team of 2 - 8 NICs connect to the same switch. Only 1 NIC is active and others on standby. Use one common IP address and MAC address. A 'failed' active adapter will pass its MAC and IP address to the standby adapter. As the name implied, it only protects against single NIC failure.

2) Switch Fault Tolerance (SFT)

Similar to AFT above, except that NICs are connected to different switches and supports only 2 NICs. In addition to single adapter protection, it protects against single switch failure.

3) Adaptive Load Balancing (ALB)

2 - 8 NICs can be teamed. One NIC receives and all ports transmit using one IP address and multiple MAC addresses. With Receive Load Balancing (RLB) enabled, more than one NIC (of the fastest speed) can receive traffic in load-balancing mode. Work with 1 or more switches.

4) Virtual Machine Load Balancing (VMLB)

Support multiple Hyper-V VMs on a team of 2 - 8 NICs. It provides transmit and receive traffic load balancing across multiple Virtual Machines (VM) bound to the common team interface. In a VMLB team, each VM is associated with one team member for its TX and RX traffic. If only one virtual NIC is bound to the team, or if Hyper-V is removed, then the VMLB team will act like an AFT team.

5) Link Aggregation

In Cisco term, it is known as EtherChannel where 2 - 8 NICs are teamed to the same switch. It uses a common virtual MAC address and IP address. It supports load-balancing across the team members.

NIC Teaming Configuration

Configuring Intel NIC teaming is a pretty straightforward task once you understand the basic network concept. For step-by-step, see this YouTube demo. Nevertheless, it is important to note that Microsoft (see KB254101 and KB968703) does not support vendor specific NIC teaming. If you raise support issues and NIC teaming is in suspect, MS may ask that the NIC teaming be removed.

In my next posting, I would talk about using Microsoft Windows 2008 Routing (RIPv2) for network redundancy as an alternative.

Re-using old network connection name

Whenever you try to name a network connection name with the same name of once-existence old network adaptor, you would encounter this error



To use back the same name, execute the following command:

SET DEVMGR_SHOW_NONPRESENT_DEVICES=1
START DEVMGMT.MSC

On the Device Manager, click on View -> Show hidden device, uninstall the non-existence network device.