Setting Up Cisco L2 and L3 Devices with GNS3 (1.5.2) for CCNA/CCNP Preparations

See my LinkedIn post on this topic: https://www.linkedin.com/pulse/setting-up-cisco-l2-l3-devices-gns3-152-ccnaccnp-preparations-yee

Cisco Flexible Packet Matching (FPM) in 15.x

Cisco FPM on ISR router is about detecting a certain pattern (e.g. regular expression) in the payload packets before deciding whether to forward or drop it. One good example is to drop malicious packets and even Skype login that attempt to change its communicating methods over time. Your IPS signatures may not even be updated quick enough.

There is an easy-to-follow FPM guide on Getting Started with Cisco IOS Flexible Packet Matching. It even stated that almost all Cisco ISR platforms support this feature. I've learnt that only certain trains and versions can support FPM commands and they may not even be the latest versions. Use "Cisco Software Advisor" on "Feature/Software" tab to determine which IOS trains and versions support FPM. Of course, you'll need a CCO account to login.

In 15.x, there is also a change in loading FPM PHDF files. Not only you don't have to download the phdf files, there is a slight change in loading FPM PHDF files:

Router(config)#load protocol system:fpm
%Complete file name to be loaded is required

Instead, you'll have to do this

Router(config)# load fpm

Try to load bundle PHDF files ...

Then do a "show protocols phdf all" to see loaded phdf files. It should include all standard PHDFs: ether.phdf, ip.phdf, tcp.phdf, and udp.phdf. These PHDFs provide Layer 2-4 protocol definition according to Flexible Packet Matching Deployment Guide.

Nested Access Control
Cisco FPM supports nested access control policy i.e. enforce a child policy on parent policy. You can define a "class-map type stack" to check on the protocol fields and use another "class-map type access-control" to check on the payload contents. For example, you want to check for a password on the payload on protocol number 17 (UDP) on port 1234. The example config would be:

!--- Define the values to be checked on UDP header port 1234
class-map type stack match-all UDP-CHECK
 match field IP protocol eq 0x11 next UDP
 match field UDP dest-port eq 1234 next UDP

!---- Ensure the payload to contain the password string. You can also use regular expression
class-map type access-control match-all PASSWORD-CHECK
 match start UDP payload-start offset 0 size 100 string "password"

!---- Define the child policy and just log the packets if payload contains the password
policy-map type access-control CHILD-POLICY
 class PASSWORD-CHECK
   log

!---- Nested policy on Parent. Check on UDP header then the payload. Otherwise, drop the packet.
policy-map type access-control PARENT-POLICY
 class UDP-CHECK
   service-policy CHILD-POLICY
 class class-default
   log
   drop

!--- Enforce the FPM policy on the router interface
interface GigabitEthernet0/0
  service-policy type access-control output PARENT-POLICY

Cisco Multicast Expansion Table (MET) is Full

Today, some users aren't able to join multicast groups to watch their favorite video. After pinpointing to the upstream Cisco 7600 router, I noticed the following error message in the log:

 %MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum
%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1
%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)

The router is configured PIM in sparse mode and an RP manually set.

(hostname)#show platform hardware capacity multicast
L3 Multicast Resources
  Replication mode: egress
  Replication capability: Module                              Mode
                          2                                                 Egress    
                          3                                                 Egress    
                          4                                                 Egress    
                          5                                                 Egress    
  MET table Entries: Module                             Total    Used    %Used
                     3                                                65526       28        1%
                     4                                                65526       38        1%
                     5                                                32744   32744     100%

Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.

There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.

After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingress

Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.

Concept of Cisco Bridge Domain Interfaces (BDI)

Today, I came across a strange configuration on a Cisco ASR router. It's called "Bridge Domain Interfaces (BDI)". I did a search on Cisco website and the configuration looked simple. But it was short on concept explanation, which simply mentioned

"Bridge domain interface is a logical interface that allows bidirectional flow of traffic between a Layer 2 bridged network and a Layer 3 routed network traffic. Bridge domain interfaces are identified by the same index as the bridge domain. Each bridge domain represents a Layer 2 broadcast domain."

What is it used for? Why do we need it? After some thoughts and experiment, it seems to me that BDI is used to "bundle" one or more physical L2 interfaces and link it to a L3 logical interface for routing. And this L3 logical interface is the BDI. As Cisco routers won't allow you to configure IP address belonging to the same L2 subnet/domain on more than one routed interfaces, BDI is probably a workaround to overcome that limitation. It also reminds me of a routed port-channel. 

Consider the following diagram:

Both physical ports (Gi0/0/0 and Gi0/0/1) are linked to the same L2 domain (e.g. VLAN 100). 

According to Cisco, 

"An Ethernet Virtual Circuit (EVC) is an end-to-end representation of a single instance of a Layer 2 service being offered by a provider to a customer. It embodies the different parameters on which the service is being offered. In the Cisco EVC Framework, the bridge domains are made up of one or more Layer 2 interfaces known as service instances. A service instance is the instantiation of an EVC on a given port on a given router. Service instance is associated with a bridge domain based on the configuration."

I would interpret that a service instance is used to represent one L2 domain. More than 1 ports can belong to the same service instance.

Config mode:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100 # get VLAN 100 tagged traffic

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identified as interface BDI 100 in below example config

Above config would create a service instance 100 that is linked to VLAN 100 L2 domain. Standard L3 config can be performed on interface BDI 100 for routing.

interface BDI100

  vrf forwarding VPNA

  ip address 1.1.1.1 255.255.255.0

  ip ospf 1 area 0

The physical interface can even join more than 1 bridge domain (up to 4096 per router). For example, connecting to VLAN 200 (also Bridge Domain 200) as well:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identify as BDI 100 in below example config

!
 service instance 200 ethernet

    encapsulation dot1q 200

    bridge-domain 200 # identified as BDI 200 

Making Changes to MST Config with Care

Multiple Spanning Tree (MST) is great for VLAN spanning tree management and load-balancing. In concept, you can have multiple VLANs and group them up into 2 regions. Instead of managing every individual VLAN spanning tree, you would just need to manage the two MST regions regardless of the number of VLANs.

However, for two or more switches to be in the same MST region, they must have the identical MST name,

VLAN-to-instance mapping, and MST revision number. Any changes applied to existing MST configuration on one switch but not on others would cause network disruptions. To minimize such disruptions, learn to make MST changes but only commit them until all switches are 'standardized' in the new MST configuration.

This is how to enter MST configuration sub-mode on the switch:

switch# configure terminal

switch(config)# spanning-tree mst configuration

This shows how to leave MST-submode configuration on the switch without committing the

changes:

switch(config-mst)# abort

This shows how to commit the changes and leave MST configuration sub-mode on the switch:

switch(config-mst)# exit

For more information, download this Cisco "Configuring MST" doc.

Ethernet over MPLS (EoMPLS) Cisco Configuration Makes Simple

Ethernet over MPLS (EoMPLS) is part of Cisco's Any Transport over MPLS to provide L2 connectivity (pseudo-wire) over MPLS cloud. If you wish to extend EoMPLS on L3 VLAN interface (SVI-based EoMPLS or SwEoMPLS), you must have an OSM or an Enhanced FlexWAN module on the MPLS core-facing interface. Otherwise, you can configure PFC-based EoMPLS on a physical interface or sub-interface. In this example, we would use PFC-based EoMPLS on Cisco IOS 15.x, as I believe most of us won't have any special interface cards. Consider this network diagram below. We'll extend L2 connectivity on VLAN 10 to connect both servers over the MPLS cloud.

Here, we assume that basic MPLS configuration has been put in place. Configuring EoMPLS would be pretty straightforward.

On both PE1 and PE2 routers:
!

interface GigabitEthernet0/1.10
encapsulation dot1Q 10
xconnect 10.1.1.x 10 encapsulation mpls
no shut

!
Replace above 'x' with the peer PE router ID i.e. on PE1 x = 2 and on PE2 x = 1. The router ID is determined by the "mpls ldp router-id" command on the router.

On both CE1 and CE2 switches, let's assume Gi0/1 switch interface is used to connect to their respective PE routers.
!
Vlan 10
  name EoMPLS
!
interface GigabitEthernet0/1
switchport trunk encapsulation dot1Q
switchport mode trunk
!
interface GigabitEthernet0/2
description Host port
switchport mode access
switchport access vlan 10
!

To verify the EoMPLS connectivity, enter "show mpls l2transport vc" on both PE routers. The status should indicate UP. You can also perform "show spanning tree vlan 10" on both CE switches to ensure the sanity of spanning tree i.e. only one of the switches should be the root. And finally, both hosts should be able to ping each other on same IP subnet. For further details, refer to this Cisco article.

BGP does not advertise iBGP-learned routes to eBGP peers?

Theoretically, BGP does not advertise iBGP-learned routes to eBGP peers. Hence, I did not filter iBGP learned routes to my eBGP peers. I didn't realise my eBGP router did advertise these iBGP routes (not meant for transit) until our partner informed me and I did a "show ip bgp vpnv4 VRF-name neighbors 1.1.1.1 advertised-routes" to verify. So, is this theory wrong?

Well, it comes with a caveat. This theory is true only if BGP synchronisation is enabled. The rule of synchronization states that routes learned via BGP must be validated by the interior routing table before they can be advertised to remote peers. So, what does this mean? Click here for explanation.

In older Cisco IOS versions, Synchronization is enabled by default. However, it is disabled by default in Cisco IOS Software Release 12.2(8)T and later. If you do not intend for the iBGP learnt routes to be on transit among different AS, it is probably safer to enable BGP synchronization.

New Cisco L2 Catalyst 2960S supports static routes

It's long overdue! Finally, we can now configure static route on our newly shipped Cisco Catalyst 2960S L2 stackable switches. By default, you can't configure any routes on it besides the "ip default-gateway". Hence, you have to switch the sdm template from the default desktop to lanbase-routing as follows:

Switch# config t
Switch(config)# sdm prefer lanbase-routing
Switch(config)# reload

Do note that the lanbase-routing template is only supported on C2960S LAN-based image and not on LAN-lite image. For more information, refer to this Cisco documentation.

SDM Mismatch on Cisco Catalyst 3750 switches

Cisco stackable switch - Catalyst 3750 - allows you to stack different models together. For example, you can stack a 12-port SFP fiber switch (Cisco 3750G-12SD) with a 48-port 1000BaseT (Cisco 3750G-48T) switch. However, if you configure the fiber switch as master, you would see the SDM Mismatch error when you perform a "show switch" command. This is described in this Cisco article.

An extract:

The Catalyst 3750-12S switch supports desktop and aggregator Switch Database Management (SDM)templates. All other Catalyst 3750 switches support only the desktop SDM templates. All stack members use the SDM template configured on the stack master. If the stack master is using an aggregator template, only Catalyst 3750-12S switches can be stack members. All other switches attempting to join this switch stack enter SDM-mismatch mode. These switches can join the stack only when the stack master is running a desktop SDM template. We recommend that your stack master use an aggregator template only if you plan to create a switch stack of Catalyst 3750-12S switches. If you plan to have a switch stack with different Catalyst 3750 switch models, configure the stack master to use one of the desktop templates.

To use the desktop template, perform the following commands on the stack master:

1. For L2 configuration: sdm prefer vlan desktop
2. For L3 configuration (enabled with ip routing): sdm prefer routing desktop
3. Do a "show sdm prefer" to verify and reload.

Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.

Multicast for MPLS VPN Extranet

In my earlier blog post, I've mentioned briefly about Multicast VPN across MPLS VPN. In this setup, the VRF must share the same default Multicast Distribution Tree (MDT), 239.1.1.1 in this example, for the multicast VPN to form dynamically. Consider the following setup:

The source server (on PE1) is streaming on multicast mode. The VRF-Green, on the other side of PE2, would be able to join to the same multicast distribution tree (239.1.1.1) and recieve the multicast stream. For this Intranet Multicast VPN to work, the VRF-Green on both PE routers should share the basic common configuration as follows:

Router PE1 and PE2
  ip vrf VRF-Green
  mdt default 239.1.1.1
  route-target export 65001:100
  route-target import 65001:100
!
ip multicast-routing
ip multicast-routing vrf VRF-Green
ip pim rp-address 1.1.1.1
ip pim vrf VRF-Green rp-address 10.1.1.1
!
router bgp 55
  address-family ipv4 mdt
    neighbor x.x.x.x activate
    neighbor x.x.x.x send-community extended
!
  address-family vpnv4
     neighbor x.x.x.x activate
     neighbor x.x.x.x send-community extended
!

However, the VRF-Blue (on PE2) won't be able to join the same MDT, as Cisco routers do not support multiple duplicate default MDT across VRFs. However, there is a workaround for this Extranet setup. Do note that data flow in MPLS VPN happens in label-exchange. Create another VRF-Green on the source PE1 and import the multicast traffic over to VRF-Blue on PE2. There are 2 options of importing, either import it on the source PE or on the reciever PE. I observed that the latter reciever PE option is more reliable if there are more than 1 reciever PE routers.

Configuring the Receiver MVRF on the Source PE (Option 1)
On PE1, add "route-target import 65001:100" on VRF-Blue, enable multicast routing "ip multicast-routing vrf VRF-Blue", and specify the same Rendezvous Point (RP) as VRF-Green for VRF-Blue using "ip pim vrf VRF-Blue rp-address 10.1.1.1".

For this setup, assign a new default MDT group for VRF-Blue on both PE routers (239.2.2.2 in this example). Import the label on VRF-Green over (route-target import 65001:100) as shown in above diagram. In this way, two multicast distribution trees (MDTs) can be created from the same source server. When you perform a "show ip mroute vrf VRF-Green" on PE1, you'll see Extranet receivers in VRF-Blue.

Configuring the Source MVRF on the Receiver PE (Option 2)
Another option is to do the import on the reciever PE (PE 2 in this example). As mentioned earlier, I observed that the stream is slightly more stable if there is more than 1 reciever PE. You'll need to do the import on every reciever PE.

Troubleshooting
Ensure RPF and multicast routing is sound for GRT level on all PE and P routers, as well as VRF level for all PE. In this example (some outputs truncated):
GRT level
#sh ip mroute
(10.10.10.1, 239.1.1.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Port-channel1, RPF nbr 10.10.0.1
  Outgoing interface list:
    Port-channel2, Forward/Sparse, 00:54:29/00:03:15

VRF level
#sh ip mroute vrf VRF-Green
(192.168.1.1, 239.23.25.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Tunnel1, RPF nbr 192.168.0.1
  Outgoing interface list:
    VLAN123, Forward/Sparse, 00:54:29/00:03:15

For further details, refer to the Cisco Multicast VPN Extranet support.

Multipath with Redundant Routers
If you have redundant paths with some load splitting or sharing, you may have to consider enabling multipath options using following steps on every router:

1. enable
2. configure terminal
3. ip multicast multipath s-g-hash next-hop-based
4. ip multicast vrf vrf-Name multipath s-g-hash next-hop-based
5. end
6. show ip rpf source-address group-address
7. show ip route ip-address

Reference: Load Splitting IP Multicast Traffic over ECMP

Load Sharing in MPLS VPN with Route Reflector

No matter what I did according to this Cisco article, I simply can't do load-sharing among the PE routers within our enterprise MPLS VPN. The central Route Reflector (RR) simply refuses to send more than 1 identical route, even though I have multiple PE routers attached to that route.

Subsequently, I come across this excellent MPLS VPN blog. It explains that there is no way to change the route reflector behavior.  The workaround is to simply assign different Route Distinguisher (RD) values for the same VRF on different routers. To enable route propagation, ensure that you export and import common Route Target (RT) across the PE routes. Finally, enable "maximum-paths eibgp " on each vpnv4 address family under the router bgp process, so as to inject more than one bgp route into the routing table.

And it works magic!

Example configuration according to above diagram

On Router PE A

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE B

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE C

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE D

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

Using Anycast RP for Video Multicasting

In my earlier post on Bridging Dense-Mode to Sparse-Mode PIM, I mentioned about using Sparse-Mode Anycast Rendezvous Point (RP). But what is it about?

First of all, let's understand what Sparse-Mode Rendezvous Point is. For all multicast PIM, there is always at least one source (which is typically a video streaming server like the Microsoft Windows Media Services) and many client receivers. Instead of sourcing from the streaming server directly (as in the case of Source-Specific Multicast), multicast sources and receivers must register with their local rendezvous point (RP). So think of RP like a common meeting point for all sources and receivers. It is also known as a Shared Tree multicast model. In Cisco, there are a few configuration models for RP, including Manual RP (i.e. "hard-coding" of RP in every router), Auto-RP (i.e. elect one of many RP candidates), bootstrap RP (similar to Auto-RP) and Anycast RP.

Most RP models support only a single active RP at any one time. Only the Anycast RP model provides load sharing and redundancy in the multicast network. Anycast RP allows two or more rendezvous points (RPs) to share the load for source registration and the ability to act as hot backup routers for each other. Two or more RPs are configured with a common IP address on loopback interfaces with a 32-bit mask, making it a host address. All other routers are statically configured to map the RP to this common address and are likely to be linked up to the nearest RP. The protocol that links up multiple RPs is known as the Multicast Source Discovery Protocol (MSDP).

Static Anycast RP is also relatively easy to configure and troubleshoot. Consider the following example from Cisco.com:

As you can see from above example, both RP1 and RP2 share the common IP address of 10.0.0.1 on loopback 0 interface. MSDP is enabled to peer both RPs on the respective loopback 1 interfaces using the ip msdp commands. All other routers (the 2 routers at the bottom) just statically map the RP to this common IP address using the ip pim rp-address 10.0.0.1 command. Of course, you have to ensure that the loopback addresses are routable within your domain, which can be easily achieved in any dynamic IGP, such as OSPF.

Troubleshooting Summary

For troubleshooting, the most common failure is the breaking of Reverse Path Forwarding (RPF), which is a mechanism for the receiving routers to determine the best path to the source or the RP in this case. The most common omission is the missing  ip pim sparse-mode or ip pim sparse-dense-mode on all router interfaces.

After which still fail, have a client join to the multicast address. Alternatively,  you may simulate the joining by having the client's next-hop L3 access switch or router to join the multicast group using ip igmp join-group 239.1.1.1 (replace this address with the actual multicast address configured on your streaming server) on the receiver's interface level. Trace on every router from the receiver to the source using show ip mroute command (with vrf option in MPLS VPN or VRF-lite situation). Check for broken reverse path (.e.g. take note of missing RPF neighbor with Null incoming interface).

#show ip mroute 239.1.1.1
.....
(10.1.12.4, 239.1.1.1), 00:00:01/00:02:58, flags:
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/0, Forward/Dense, 00:00:02/00:00:00
Vlan111, Forward/Dense, 00:00:02/00:00:00

Take note of above output, if you see that the incoming interface is null and the neigbour is empty on a router (bold red), it usually means a missing "ip pim" on some interfaces or an issue on the underlying unicast routing issue. If it's latter, perform standard routing troubleshooting on the underlying IGP or static routes.

There is this good blog post that further elaborates the detailed steps on RPF troubleshooting.

Faster OSPF Convergence using iSPF

According to this Cisco config guide, OSPF uses Dijkstra's SPF algorithm to compute the shortest path tree (SPT). During the computation of the SPT, the shortest path to each node is discovered. The topology tree is used to populate the routing table with routes to IP networks. When changes to a Type-1 or Type-2 link-state advertisement (LSA) occur in an area, the entire SPT is recomputed.

In many cases, the entire SPT need not be recomputed because most of the tree remains unchanged. Incremental SPF (iSPF) allows the system to recompute only the affected part of the tree. Recomputing only a portion of the tree rather than the entire tree results in faster OSPF convergence and saves CPU resources. Note that if the change to a Type-1 or Type-2 LSA occurs in the calculating router itself, then the full SPT is performed. Incremental SPF is scheduled in the same way as the full SPF. Routers enabled with incremental SPF and routers not enabled with incremental SPF can function in the same internetwork.

Given only pros and not cons, we should enable iSPF by default. iSPF can be easily enabled using ispf command under each router ospf process.

1. router ospf 1
2. ispf
3. !

To verify:

1. show ip ospf 1 | inc SPF
2. ........
3. Incremental-SPF enabled 
4. .......

VRF-aware Dynamic Multipoint VPN

Following on my previous post on VRF-aware Multipoint GRE, you can further protect the mGRE tunnels with IPSec easily.

1. On all Routers


2. !


3. crypto keyring ciscokey vrf outer


4. pre-shared-key address 172.16.0.0 255.255.0.0 key cisco123


5. !


6. crypto isakmp profile isaDMVPN


7. keyring ciscokey


8. match identity address 172.16.0.0 255.255.0.0 outer


9. !


10. crypto ipsec transform-set tfDMVPN esp-aes esp-sha-hmac


11. mode transport


12. !


13. crypto ipsec profile proDMVPN


14. set security-association lifetime seconds 900


15. set transform-set tfDMVPN set isakmp-profile isaDMVPN


16. !


17. interface Tunnel1


18. ip vrf forwarding inner


19. tunnel protection ipsec profile proDMVPN #apply protection on tunnel

To verify, perform the following commands and check the status in bold:

1. Router1#sh crypto isakmp sa


2. IPv4 Crypto ISAKMP SA


3. dst src state conn-id slot status


4. 172.16.1.1 172.16.1.2 QM_IDLE 1001 0 ACTIVE


5. ......


6. Router1#sh crypto session


7. Crypto session current status


8. Interface: Tunnel1


9. Profile: isaDMVPN


10. Session status: UP-ACTIVE


11. ......

If you couldn't obtain the above results, perform further troubleshooting using "debug crypto isakmp" and "debug crypto ipsec". Most of the time, the issues are related to isakmp issues and authentication key. Check that all the parameters (esp pre-shared key) can match.

VRF-aware Multipoint GRE Tunnel

GRE tunnel is especially useful to encapsulate network traffic (including IGP routing) separate from the underlying infrastructure. One useful way is to create a logical Out-of-Band (OOB) management interface for network management on another VRF. The default point-to-point GRE tunnels are easy to setup when you have few remote sites to connect to. It will become less scalable, as the number of sites grow. To overcome it, consider point-to-multipoint tunnel instead. Instead of having multiple tunnels on the Hub router, you can have just one tunnel to connect to all spoke sites. And you don't have to manually map all the spoke destination addresses, as all the spoke routers will register dynamically with the Hub router. Consider the following diagram (click to enlarge):
Assuming that the outer VRF (including the loopback interfaces) is already made routable (e.g. OSPF, BGP etc) within this network. We are setting up another VRF on the inner for network management purposes.

1. On Hub CE Router A:
2. interface Tunnel1
3. ip vrf forwarding inner
4. ip address 192.168.1.1 255.255.255.0
5. ip nhrp authentication cisco #ensure matching key for all spokes
6. ip nhrp map multicast dynamic
7. ip nhrp network-id 123 #also ensure network-id can match
8. ip ospf network broadcast
9. ip ospf priority 10
10. tunnel source Loopback0 #note that the destination is not defined
11. tunnel mode gre multipoint
12. tunnel key 123 # and the tunnel key as well
13. tunnel vrf outer # create a tunnel on the outer vrf
14. !
15. router ospf 120 vrf inner
16. network 172.16.1.1 0.0.0.0 area 0
17. network 192.168.1.0 0.0.0.255 area 0

1. On spoke router B
2. interface Tunnel1
3. ip vrf forwarding inner
4. ip address 192.168.1.2 255.255.255.0
5. ip nhrp authentication cisco
6. ip nhrp map 192.168.1.1 10.1.1.1 #map inner vrf to outer vrf on hub router
7. ip nhrp map multicast 10.1.1.1 #register with nhrp hub using multicast
8. ip nhrp network-id 123
9. ip nhrp nhs 192.168.1.1 #define hub router as next hop
10. ip ospf network broadcast
11. ip ospf priority 0
12. tunnel source Loopback0
13. tunnel mode gre multipoint
14. tunnel key 123
15. tunnel vrf outer
16. !
17. router ospf 120 vrf inner
18. network 172.16.1.2 0.0.0.0 area 0
19. network 192.168.1.0 0.0.0.255 area 0

1. On spoke router C
2. interface Tunnel1
3. ip vrf forwarding inner
4. ip address 192.168.1.3 255.255.255.0
5. ip nhrp authentication cisco
6. ip nhrp map 192.168.1.1 10.1.1.1 #map inner vrf to outer vrf on hub router
7. ip nhrp map multicast 10.1.1.1 #register with nhrp hub using multicast
8. ip nhrp network-id 123
9. ip nhrp nhs 192.168.1.1 #define hub router as next hop
10. ip ospf network broadcast
11. ip ospf priority 0
12. tunnel source Loopback0
13. tunnel mode gre multipoint
14. tunnel key 123
15. tunnel vrf outer
16. !
17. router ospf 120 vrf inner
18. network 172.16.1.3 0.0.0.0 area 0
19. network 192.168.1.0 0.0.0.255 area 0

If you have a second Hub router on the headend, you can setup another multippoint tunnel for redundancy like the following diagram:
Nevertheless, the newly created overlay VPN remains in plain. You may also wish to protect it using IPSec. Cisco calls this combination of IPSec and Multipoint GRE as "Dynamic Multipoint VPN" or "DMVPN", which I should blog about it in my next post.

Cisco Performance Routing (PfR)

For most medium-to-large networks, you would usually have more than 1 ISP to connect the enterprise network to all the remote sites. Traditional routing (such as OSPF, BGP etc) could route network traffic through a preferred path or load-balance among different paths. It can also automatically change path when a link is dead.

However, traditional routing alone can't detect and take actions on links that are "half-dead" e.g. congested, flappy link etc. To overcome it, Cisco introduces Performance Routing (PfR) or f.k.s Optimized Edge Routing (OER) and add intelligence bits into the classical IP routing. There are two components to PfR: a Master Controller (MC) and a Border Router (BR). In a typical installation, there is one MC and one or more BRs. In many smaller designs, you can have the MC and BR on the same physical router.

The MC is the brain of PfR. The MC receives performance data from the BRs, compares it against the configured policy, selects the best route for the data and sends commands to the BR to forward traffic to that path.

The BR is the foot soldier of PfR. It collects the performance data and sends it to the MC. The MC compares the data against the configured performance policy. If it is out of policy, the MC sends commands to the BR to change the data path. PfR can monitor link performance based on passive mode (using netflow) and active mode (IP SLA).

There is this good wiki article that explains the fundamental of PfR.

Route filtering using route tags

In enterprise routing, route-filtering is often used to prevent routing loops and sometimes for security reasons. Instead of solely relying on ip access-list and addresses, route filtering can also be performed by route tagging. In fact, this method is more scable for a larger network when managing access-lists can be a challenge over a large number of routers. Consider this corporate network (see below pic). The corporate has 3 remote sites with IP subnets of 1.1.1.0, 2.2.2.0 and 3.3.3.0 respectively. You have a corporate policy that states Network A should link to all 3 remote sites via ISP X. Network B should link to the first 2 remote sites via leased lines and the last remote site via ISP X only. Network A is peered with ISP X on eBGP. IGP between internal networks is OSPF and remote sites via leased line is RIP. To implement such routing policy using route-tag:

1. Router A


2. access-list 1 permit 1.1.1.0 255.255.255.0


3. access-list 1 permit 2.2.2.0 255.255.255.0


4. access-list 2 permit 3.3.3.0 255.255.255.0


5. !


6. route-map route-tag permit 10 ‌


7. match ip address 1 ‌


8. set tag 111 --tag the 1st two remote sites with 111


9. !


10. route-map route-tag permit 20 ‌


11. match ip address 2 ‌


12. set tag 222 -- tag the 3rd remote site with 222


13. !


14. route-map route-tag permit 30 -- without this, all other routes will be dropped


15. !


16. router ospf 1 ‌


17. redistribute bgp 65001 subnets route-map route-tag -- redistribute ISP routes into IGP


18. ...


19. ...


20. Router B


21. route-map tag-filter deny 10 ‌


22. match tag 111 -- filter off sites with tag 111


23. !


24. route-map tag-filter permit 20 ‌


25. match tag 222 --permit only sites with tag 222


26. !


27. router ospf 2 ‌


28. distribute-list route-map tag-filter in

To verify, perform the necessary "show ip route" commands on both router A and B to ensure the route entries are in order. Do note that tagging does not work with BGP. The alternative in BGP is to use community string in AA:NN format (e.g. 100:300). For the adverting routers (typically on customer edge), use "set community" in place of "set tag" in the route-map statement. For the recieving routers (typically on provider edge), use "ip community-list" to describe the community string and "match community". For further example on using BGP community, see this Cisco example.

Verifying Cisco IOS Image Checksum

It's always advisable to check the MD5 checksum after you downloaded the new IOS image and again after you uploaded the new image to the Cisco device. Otherwise, your routers may not even able to boot up with corrupted images. There are free MD5 checksum program available on the Internet. One such program is the MD5 Checker. And it's also probably good idea to store the MD5 value alongside with the image.

Once you have uploaded the new image and before you reload the router, run this command:

#verify /md5 ‹ ios image location ›
example: verify /md5 flash:c1841-adventerprisek9-mz.124-25e.bin

Compare the output value with the MD5 sum that you noted earlier.

Dell PowerConnect VLAN Interoperability with Cisco

According to this document, Cisco catalyst switches and Dell PowerConnect switches (in fact, most other non-Cisco switches e.g. Juniper etc) are not compatible on VLAN trunking i.e. single physical link that carries multiple VLANs. This is because Dell and Cisco devices have different default mechanisms for dynamic exchange of VLAN information. Cisco Catalyst switches’ default mechanism is the VLAN trunking protocol(VTP). In contrast, Dell PowerConnect switches use the GARP VLAN registration protocol (GVRP) for dynamic exchange of VLAN configuration information. Because the Dell and Cisco switches use different protocols by default, no exchange of VLAN control traffic will take place – and thus no intra-VLAN traffic will flow between the Dell and Cisco switches. By disabling VTP and enabling GVRP on the Cisco switch, it is possible to exchange intra-VLAN data and control information in mixed Dell-Cisco environments.

I have just recieved confirmation from Cisco saying this:

Unfortunately, GVRP is not supported on Catalyst 3560 or 3750 switches. GVRP is only supported on the CatOS releases on Cat4000 and Cat6500 platfroms (on select releases - Only 6000s, 5000s and 4000 switches running CatOS software support this feature.)

If you connect a 3560 or 3750 switch to a device that supports GVRP you will see unsupported messages on the switch, telling you that the device itself is not able to process the GVRP information from the hosts/neighbours and therefore it drop it.

Nevertheless, GVRP is supported on IOS Router (cGVRP):
http://www.cisco.com/en/US/partner/docs/ios/12_2sr/12_2srb/feature/guide/srbcgvrp.html
What's the hell?! CatOS is already obsolete. And IOS routers being at L3 do not even need to pass VLAN trunking information. Does it mean Cisco lock-in?