Comparision between iSCSI and FCoE on Cisco Nexus

There are much confusions and debates over the choice between iSCSI and FCoE (Fiber Channel over Ethernet) for SAN storage choice, especially when you have invested on the Cisco Nexus data center infrastructure. Both choices are well supported by Nexus and both can definitely run concurrently together with the data network traffic. On top of that, you can even run both Jumbo and non-Jumbo frames without any loss in packets and performance! This truly fulfill the promise of "Unified Fabric Data Center".

The main advantage of iSCSI is its low-cost and its ease of implementation. You don't need any additional hardware on your existing servers and you can run it off on any server-class Gigabit Ethernet NICs that support TCP offload (even though it's non-mandatory). Because iSCSI runs on top of TCP/IP, you may suffer some slight deterioration in performance due to its inherent latency, especially when you have to route the traffic over L3 hops.

FCoE, on the hand, is a replicate of existing FC technologies (except the physical cables, it simply replaces the Fibre Channel physical layer with 10GbE). If you do own existing FC infrastructure (i.e. Cisco MDS) and intend to "migrate", "converge" or "consolidate" into Ethernet infrastructure, FCoE would be the choice. You can simply map VSAN into VLAN on the Lossless Ethernet-based Nexus for inter-operability. Typically, you would need Converged Network Adapters (CNAs) on the servers, which contain both Fibre Channel Host Bus Adapter (HBA) and Ethernet Network Interface Card (NIC) functionality on the same adapter card. CNAs have one or more physical Ethernet ports, which facilitate transition and migration from FC to FCoE.

Below is a good comparision table between iSCSI and FCoE taken from this NetApp site.

Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.

Configuring Nexus 5000 with Nexus 2000 Fabric Extenders

Top-of-rack switching is commonly deployed for high port-density data centers. Switches are mounted at the top of each rack that makes cabling looks neat and tidy. However, it leads to switch managability issues dealing with multiple spanning-trees when you have hundreds of layer 2 switches - not to mention firmware upgrading. To resolve these issues, we are using Nexus 5000 with multiple fabric extenders (Nexus 2000). The N2K are just like the line cards to a chassis switch e.g. Catalyst 6500, except that N5K doesn't peform L3 functions like IP routing. This setup allows you to manage the whole bunch of switches as a single switch distributed all over the data center as shown below.

Between N5K and N2K, you may connect upto 4 x fiber links . That is giving you up to 40Gbps uplink per extender. As the Nexus are running on Cisco NX-OS, most switching commands are similar to the tradition Cisco IOS. However, the setup configuration is different. You need to perform the following steps:

1) Create a virtual fex chassis
switch(config)# fex 117
! set the number of links from N5K to N2K
switch(config-fex)# pinning max-links 1 (you may set up to 4 links)
2) Associate the N2K extenders to the fex chassis
switch(config)# interface e1/17
switch(config-if)# switchport mode fex-fabric
switch(config-if)# fex associate 117
3) To verify
switch# sh int e1/17 fex-intf

You may now configure the individual switch ports on the N2K extenders like normal Cisco switch ports. Unlike other Cisco switches, most switching features are not enabled by default. You'll have to turn them on manually using the "feature" command. For example, if you wish to configure EtherChannel, you have to enable LACP using "feature lacp" command.