Internet Load Balancing for Dual WAN Links to ISP

Recently, the WAN upgrade to our ISP is complete with dual redundant paths. We were asked if we could allow our Internet surfing users utilizing both links as much as possible instead of leaving one link to be idle most of the time. At the same time, it must not break the existing path redundancy. Since BGP rules Internet routing, we are using it to our advantage.

Below is the simplified network diagram to keep this discussion simple. (Click to enlarge)


Our 2 routers and the provider's routers are peered in full mesh eBGP. We advertised our public IP subnet (say 160.1.1.0/29) to the world via the 2 ISP routers (i.e. ISP R1 and ISP R2). By default in BGP, only 1 best route (i.e. default route) from either ISP R1 or ISP R2 is chosen as the path to the Internet. To influence the routing behavior, AS path prepend is used to influence inbound traffic and local preference to outbound traffic. As for load-balancing, whatever traffic that entered via R1 will route through ISP R1 and the similar applies to R2. This is our strategy for ISP link load-balancing:

1. We further break our public IP subnet into 2 halves i.e. 160.1.1.0/30 and 160.1.1.4/30 and advertise them via both R1 and R2.
2. On R1, we prepend AS path on advertised route 160.1.1.4/30 to make it less desirable for inbound traffic to use this route via R1. On R2, we prepend AS path on route 160.1.1.0/30.
3. On R1, higher local preference is set for default route (0.0.0.0/32) advertised from ISP R1. Hence, ISP R1 will be the preferred next-hop for all outbound Internet traffic entered via R1. As for R2, the next preferred next-hop will be ISP R2.
4. In summary, the path will become R1 <-> ISP R1 and R2 <-> ISP R2. We influence inbound traffic by making the other route less attractive and outbound traffic by making the route more attractive. If either R1 or R2 link were down, the remaining active link will take over all the traffic.

As for load-balancing between our routers (R1 & R2), it is more straightforward. Have both routers to advertise default route (on same metric) into the IGP (e.g. OSPF or RIP) by using "default information-orginate" router command. Alternatively, you may prefer GLBP (Gateway Load Balancing Protocol) for multiple clients.

The diagram (courtesy from my colleague MT) below illustrates the BGP load-balancing concept described above. (Click to enlarge)


Any sample configuration? Here you are:

On R1:
router bgp 65001
bgp router-id 172.16.1.1
bgp log-neighbor-changes
no auto-summary
neighbor 172.16.1.3 remote-as 65002
neighbor 172.16.1.3 route-map R1-ISPR1-MAP out # apply AS path prepend
neighbor 172.16.1.3 activate
neighbor 172.16.1.4 remote-as 65002
neighbor 172.16.1.4 route-map ISPR1-R1-MAP in # set higher local preference
neighbor 172.16.1.4 activate
no synchronization
network 160.1.1.0 mask 255.255.255.252 # route advertisement
network 160.1.1.4 mask 255.255.255.252
!
# exact routes must exist before they can be advertised in eBGP!
# since we are using NAT, just create some "phantom" routes
ip route 160.1.1.0 255.255.255.252 null0
ip route 160.1.1.4 255.255.255.252 null0
!
!
# Use NAT overload for internal users accessing Internet
ip nat pool INET_POOL 160.1.1.1 160.1.1.1 netmask 255.255.255.252
ip nat inside source list INSIDE_VLAN pool INET_POOL overload
!
ip access-list standard INSIDE_VLAN
permit 192.168.2.0 0.0.0.63
!
access-list 11 permit 160.1.1.0 0.0.0.3
access-list 12 permit 160.1.1.4 0.0.0.3
!
#make certain route advertised by this router less desirable
#to influence inbound traffic
route-map R1-ISPR1-MAP permit 10
match ip address 12
set as-path prepend 65001 65001 65001
!
route-map R1-ISP1-MAP permit 20
match ip address 11
!
#prefer default route from specific ISP router to influence outbound traffic
route-map ISPR1-R1-MAP permit 10
set local-preference 200

On R2:
router bgp 65001
bgp router-id 172.16.1.2
bgp log-neighbor-changes
no auto-summary
neighbor 172.16.1.3 remote-as 65002
neighbor 172.16.1.3 route-map ISPR2-R2-MAP in
neighbor 172.16.1.3 activate
neighbor 172.16.1.4 remote-as 65002
neighbor 172.16.1.4 route-map R2-ISPR2-MAP out
neighbor 172.16.1.4 activate
no synchronization
network 160.1.1.0 mask 255.255.255.252
network 160.1.1.4 mask 255.255.255.252
!
ip route 160.1.1.0 255.255.255.252 null0
ip route 160.1.1.4 255.255.255.252 null0
!
!
ip nat pool INET_POOL 160.1.1.5 160.1.1.5 netmask 255.255.255.252
ip nat inside source list INSIDE_VLAN pool INET_POOL overload
!
ip access-list standard INSIDE_VLAN
permit 192.168.2.0 0.0.0.63
!
access-list 11 permit 160.1.1.0 0.0.0.3
access-list 12 permit 160.1.1.4 0.0.0.3
!
route-map R2-ISPR2-MAP permit 10
match ip address 11
set as-path prepend 65001 65001 65001
!
route-map R2-ISP2-MAP permit 20
match ip address 12
!
route-map ISPR2-R2-MAP permit 10
set local-preference 200