Today, some users aren't able to join multicast groups to watch their favorite video. After pinpointing to the upstream Cisco 7600 router, I noticed the following error message in the log:
%MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum
%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1
%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)
The router is configured PIM in sparse mode and an RP manually set.
(hostname)#show platform hardware capacity multicast
L3 Multicast Resources
Replication mode: egress
Replication capability: Module Mode
2 Egress
3 Egress
4 Egress
5 Egress
MET table Entries: Module Total Used %Used
3 65526 28 1%
4 65526 38 1%
5 32744 32744 100%
Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.
There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a
pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.
After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)#
mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)#
ip multicast hardware-switching replication-mode ingress
Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.