Recently, I installed a 1000M Shanghai Mobile broadband in the place I am currently staying.
In order to make full use of the new broadband and the existing Shanghai Telecom broadband,
it’s necessary to use some magic to utilize both upstreams.
Here, we will take the example of the CCR2004-16G-2S+ router to explain how to optimize network access using BGP full table.
Our target strategy is very clear:
- For IP addresses targeting China Telecom (AS4134, AS4812), use China Telecom as default gateway.
- For all remaining non-China ASNs, use a special gateway within the local network tunneling all traffic to one server located in Hong Kong.
- The rest should go through China Mobile.
Basic network configuration
First, we need to connect our router to the internet. Assuming the China Telecom network is connected to ether1 and the China Mobile network is connected to ether2, we can create two PPPoE clients like this:
1
2
|
/interface/pppoe-client/add name="pppoe-chinanet" interface=ether1 profile=default add-default-route=no dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
/interface/pppoe-client/add name="pppoe-chinamobile" interface=ether2 profile=default add-default-route=no dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
|
The part about the PPPoE account and password is omitted above.
You can fill it in as needed.
It is worth noting that the newly created PPPoE connection does not use the default gateway.
Here, we can add a static route as the default gateway configuration. Assuming we plan to use mobile broadband as the default upstream connection:
1
|
/ip/route/add dst-address=0.0.0.0/0 routing-table=main gateway=pppoe-chinamobile distance=100
|
At this point, our router should be able to access the network correctly. Let’s use traceroute to verify it.
1
2
3
4
5
6
7
8
9
10
11
12
|
[admin@shanghai-router] > /tool/traceroute 223.5.5.5
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
# ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV
1 <redacted> 0% 9 3.9ms 4.7 3.6 8.9 1.9
2 <redacted> 75% 9 timeout 3.7 3.6 3.7 0.1
3 <redacted> 12.5% 8 timeout 3.7 3.6 3.8 0.1
4 <redacted> 0% 8 12.9ms 7.8 3.9 20.3 5.5
5 100% 8 timeout
6 116.251.116.93 0% 8 6.2ms 12.9 5.6 38.3 10.6
7 100% 8 timeout
8 100% 8 timeout
9 223.5.5.5 0% 8 5.5ms 5.4 5.2 5.6 0.1
|
At the same time, we can also test the PPPoE connection of the China Telecom network by specifying the interface in parameters:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[admin@shanghai-router] > /tool/traceroute 223.5.5.5 interface=pppoe-chinanet
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
# ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV
1 <redacted> 7.7% 13 3.8ms 4.7 1.8 7.7 1.8
2 <redacted> 0% 13 10.3ms 8.7 4.6 18.4 3.2
3 <redacted> 66.7% 13 timeout 6.6 3.9 14.2 4.4
4 <redacted> 66.7% 12 timeout 5.5 3.8 8.7 1.9
5 <redacted> 0% 12 3.2ms 6.3 3.1 25.3 6.7
6 <redacted> 0% 12 5.8ms 18.2 5.5 150.8 40
7 100% 12 timeout
8 100% 12 timeout
9 100% 12 timeout
10 223.5.5.5 0% 12 5.4ms 5.5 5.3 5.8 0.2
|
Of course, we also need to create some NAT rules in /ip/firewall/nat, so that devices in our LAN can also access the internet. This part is relatively basic, so I won’t go into detail.
Obtain realtime BGP full routing table
If you are a BGP player, you can totally skip this part, but if you are not familiar with BGP or don’t have an ASN,
you may encounter some difficulties.
Here, you can acquire the complete BGP routing table by purchasing the cheapest Vultr server,
and it does not require you to have your own ASN.
It offers great value for money.
There are many online resources available that provide step-by-step instructions on how to perform the related operations, you can google it by yourself.
After completing the operations on Vultr, we can follow the official guide and establish a BGP session with Vultr to obtain full routing table data. It is very easy. The key configurations are as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
|
protocol device {}
protocol kernel kernel4 {
ipv4 {
import none;
export where source != RTS_DEVICE && proto != "bgp_vultr_v4";
};
}
protocol kernel kernel6 {
ipv6 {
import none;
export filter {
if source = RTS_DEVICE then reject;
if proto = "bgp_vultr_v4" then reject;
reject;
};
};
}
protocol static static_bgp_neighbor_v6 {
ipv6;
route 2001:19f0:ffff::1/128 via fe80::fc00:4ff:fe08:ee5f % enp1s0;
}
protocol static static_bgp_neighbor_v4 {
ipv4;
route 169.254.169.254/32 via a.b.c.1 % enp1s0;
}
protocol bgp bgp_vultr_v4 {
local as 142641;
neighbor 169.254.169.254 as 64515;
source address a.b.c.d;
multihop 2;
password "xxxxxxxx";
ipv4 {
import all;
export none;
};
}
protocol bgp bgp_vultr_v6 {
local as 142641;
neighbor 2001:19f0:ffff::1 as 64515;
source address 2001:db8::1;
multihop 2;
password "xxxxxxx";
ipv6 {
import all;
export none;
};
}
|
If everything goes well, we can use the command show protocol in birdc to see that our session status is already “Established”.
1
2
3
|
Name Proto Table State Since Info
bgp_vultr_v4 BGP --- up 2023-01-21 Established
bgp_vultr_v6 BGP --- up 2023-01-21 Established
|
We can also use the “show route for” command to view specific forwarding rule, for example:
1
2
3
4
5
6
7
8
9
10
11
12
|
bird> show route for 1.1.1.1 all
Table master4:
1.1.1.0/24 unicast [bgp_vultr_v4 2023-08-01 from 169.254.169.254] * (100/?) [AS13335i]
via 45.32.28.1 on enp1s0
Type: BGP univ
BGP.origin: IGP
BGP.as_path: 64515 20473 13335
BGP.next_hop: 169.254.169.254
BGP.local_pref: 100
BGP.aggregator: 10.34.7.24 AS13335
BGP.community: (20473,200) (64515,44)
BGP.large_community: (20473, 200, 13335) (20473, 200, 45686)
|
Thus we are now ready with our own BGP speaker.
Establish a tunnel between the BGP feeder and the router.
Home broadband usually does not have a static IP address.
To minimize configuration changes, the simplest way is to set up a tunnel between the router at home and a server with full BGP table, and WireGuard can be used for this purpose.
Its advantage is that it has native support in newer kernels and RouterOS.
Additionally, it uses UDP protocol, which makes it convenient to use various IEPL/IPLC forwarding services to improve reliability.
On the feeder, let’s create a new VPN interface with the following configuration:
1
2
3
4
5
6
7
8
9
|
[Interface]
PrivateKey = xxx
Address = 10.22.33.1/24, fd34:38a:f295::1/64
MTU = 1400
ListenPort = 114514
[Peer]
PublicKey = yyy
AllowedIPs = 10.22.33.2/32, fd34:38a:f295::2/128
|
Then, create a WireGuard interface on RouterOS and peering it with our BGP speaker:
1
2
3
4
5
|
/interface/wireguard/add name="wg-bgp-speaker" private-key="xxx" mtu=1400
/interface/wireguard/peers/add endpoint-address=172.16.1.1 endpoint-port=3000 interface=wg-bgp-speaker public-key="qzgKhWy8u0DBvXcrhfFDYPwKzKipWzCsxR5ecNIYZFs=" persistent-keepalive=60 allowed-address=10.22.33.1/32,fd34:38a:f295::1/128
/ip/address/add interface=wg-bgp-speaker address=10.22.33.2/24
/ipv6/address/add interface=wg-bgp-speaker address=fd34:38a:f295::2/64
|
So we have created a direct connected tunnel, which can only allow our router and server to communicate point-to-point. But that’s enough.
Prepare the routing filters
According to the rules mentioned earlier, we can prepare the routing filters on the RouterOS side.
In this filter, we directly choose the gateway used by this route based on the end of the AS path. Taking IPv4 as an example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
if (dst-len == 0) {
reject
}
set distance 50;
if (not bgp-as-path [[:china_asn_list:]]$) {
set gw 198.18.1.1;
set gw-interface ipip-internet-proxy;
accept;
} else {
if (bgp-as-path 4134$|4812$) {
set gw %pppoe-chinanet;
} else {
set gw %pppoe-chinamobile;
}
}
accept;
|
There are two things that need clarification:
-
china_asn_list is an ASN (Autonomous System Number) list obtained from GitHub, which is used in mainland China. It can be maintained in /routing/filter/num-list, for example, by adding AS4134: /routing/filter/num-list/add list=china_asn_list range=4134
-
You can directly set an interface as a gateway using set gw %interface command, which allows the kernel to pay attention to which next hop to use, and it can reduce a lot of maintenance costs.
For IPv6 filter, the content remains largely the same, with only a few adjustments required for the cross-border access gateway.
Once these two filters are prepared, the final step is to bring the full table home.
Bring full-table home
The final task is to establish a BGP session between our feeder and router. First, we need to add two new protocols on the speaker.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
protocol bgp bgp_speaker_v4 {
local as 65000;
neighbor 10.22.33.2 as 65001;
source address 10.22.33.1;
passive;
ipv4 {
import none;
export where proto = "bgp_vultr_v4" && 223.5.5.5 !~ net;
};
};
protocol bgp bgp_speaker_v6 {
local as 65000;
neighbor fd34:38a:f295::2 as 65001;
source address fd34:38a:f295::1;
passive;
multihop;
ipv6 {
import none;
export where proto = "bgp_vultr_v6";
};
};
|
Then add the corresponding BGP connection on RouterOS:
1
2
3
4
5
6
7
8
9
10
11
|
/routing/bgp/connection/add remote.address=10.22.33.1/32 remote.as=65000
local.address=10.22.33.2 remote.role=ebgp
routing-table=main router-id=10.22.33.2 as=65001 multihop=yes hold-time=infinity address-families=ip
output.filter-chain=bgp-speaker-export
input.filter=bgp-speaker-v4
/routing/bgp/connection/add remote.address=fd34:38a:f295::1/128 .as=65000
local.address=fd34:38a:f295::2 .role=ebgp
routing-table=main router-id=10.22.33.2 as=65001 multihop=yes hold-time=infinity address-families=ipv6
output.filter-chain=bgp-speaker-export
input.filter=bgp-speaker-v6
|
After waiting for the entire table to enter FIB, we can start verifying the effectiveness of our traffic diversion. We will use a China Telecom/China Mobile/China Unicom/Cloudflare addresses respectively for testing.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
# China Telecom
[admin@shanghai-router] > /ip/route/print where 221.227.153.1 in dst-address and routing-table=main and !static
Flags: D - DYNAMIC; A - ACTIVE; b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb 221.224.0.0/13 pppoe-chinanet 50
# China Mobile
[admin@shanghai-router] > /ip/route/print where 120.232.236.5 in dst-address and routing-table=main and !static
Flags: D - DYNAMIC; A - ACTIVE; b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb 120.192.0.0/10 pppoe-chinamobile 50
DAb 120.232.0.0/16 pppoe-chinamobile 50
DAb 120.232.236.0/22 pppoe-chinamobile 50
# China Unicom
[admin@shanghai-router] > /ip/route/print where 103.45.78.1 in dst-address and routing-table=main and !static
Flags: D - DYNAMIC; A - ACTIVE; b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb 103.45.76.0/22 pppoe-chinamobile 50
# Cloudflare
[admin@shanghai-router] > /ip/route/print where 1.1.1.1 in dst-address and routing-table=main and !static
Flags: D - DYNAMIC; A - ACTIVE; b, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb 1.1.1.0/24 198.18.1.1%ipip-internet-proxy 50
|
In this way, our dual-line broadband IPv4 is fully utilized,
although v6 achieves traffic diversion through these methods, there are still issues because there is no NAT and devices cannot intelligently choose which prefix of the IP address to use.
The subsequent article will introduce how to solve this problem. Finally, let’s show the results:
