%MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum
%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1
%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)
The router is configured PIM in sparse mode and an RP manually set.
(hostname)#show platform hardware capacity multicast
L3 Multicast Resources
Replication mode: egress
Replication capability: Module Mode
2 Egress
3 Egress
4 Egress
5 Egress
MET table Entries: Module Total Used %Used
3 65526 28 1%
4 65526 38 1%
5 32744 32744 100%
Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.
There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.
After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingress
Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.
After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingress
Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.
Just curious as I've never ran into this issue- How many users do you think you had running multicast at that time?
ReplyDeleteMulticast PIM is designed to replicate the same packet stream on interfaces subscribing to the same group. You may have 1000 users on one VLAN or just 10 users on 10 VLAN interfaces. Hence, the number of users is really immaterial.
DeleteWhat was done in troubleshooting to show which are the users that aren't able to join multicast groups?
ReplyDeleteBardzo fajnie napisane. Jestem pod wrażeniem i pozdrawiam.
ReplyDelete