環境
※Windowサイズを確認する為、ホストA,BでTCPセグメンテーションオフロードは無効化する。
# ethtool -K enp0s25 tso off gso off
※L2スイッチのIEEE802.3Xのフローコントロールは無効にして確認。有効/無効の違いでUDPの場合、有効の方が3倍程スループットが高くなる差異が出た為。
IPSEC-VPNのTunnelのMTUは1280バイト
RT107eのデフォルト値
http://www.rtpro.yamaha.co.jp/china/support/download/manual/Rev.9.00.20/Cmdref_j.pdf
環境動作確認
ホストA→ホストBへのPINGのRTT(ICMPデータサイズ1472バイト)
# ping -s 1472 192.168.100.251 -c 1 > /dev/null 2>&1 ; ping -s 1472 192.168.100.251 -c 50 | tail -2 50 packets transmitted, 50 received, 0% packet loss, time 49071ms rtt min/avg/max/mdev = 3.309/3.533/8.347/0.705 ms
ホストA→ホストBへscpで100MBのファイル送信
# cat /tmp/100mb.file > /dev/null; scp /tmp/100mb.file ubuntu@192.168.100.251:/tmp/. 100mb.file 100% 100MB 7.4MB/s 00:13
スループット確認
ホストA→ホストBへのiperf結果
実行コマンド
以下の???の箇所を変えて複数パタン確認
# iperf3 -c 192.168.100.251 -l ??? -u -b ???M -t 30
結果
| No | 帯域(-b) | サイズ(-l) | 送信パケット状況 (Lost/Total) |
送信パケロス率 | 送信スループット | 受信パケット状況 (Lost/Total) |
受信パケロス率 | 受信スループット |
|---|---|---|---|---|---|---|---|---|
| 1 | 10Mb/s | 1252 | 0/29952 | 0% | 10 Mbits/sec | 0/29952 | 0% | 10 Mbits/sec |
| 2 | 10Mb/s | 1253 | 0/29928 | 0% | 10 Mbits/sec | 0/29928 | 0% | 10 Mbits/sec |
| 3 | 10Mb/s | 1472 | 0/25475 | 0% | 10 Mbits/sec | 0/25475 | 0% | 10 Mbits/sec |
| 4 | 70Mb/s | 1252 | 0/209659 | 0% | 70 Mbits/sec | 0/209658 | 0% | 70 Mbits/sec |
| 5 | 70Mb/s | 1253 | 0/209491 | 0% | 70 Mbits/sec | 176850/209483 | 84% | 10.8 Mbits/sec |
| 6 | 70Mb/s | 1472 | 0/178324 | 0% | 70 Mbits/sec | 146374/178303 | 82% | 12.4 Mbits/sec |
| 7 | 80Mb/s | 1252 | 0/239610 | 0% | 80 Mbits/sec | 0/239610 | 0% | 80 Mbits/sec |
| 8 | 100Mb/s | 1252 | 0/299512 | 0% | 100 Mbits/sec | 40609/299512 | 14% | 86.4 Mbits/sec |
| 9 | 100Mb/s | 1472 | 0/254748 | 0% | 100 Mbits/sec | 223833/254439 | 88% | 12.0 Mbits/sec |
| 10 | 1000Mb/s | 1472 | 0/2241581 | 0% | 880 Mbits/sec | 845292/856348 | 99% | 4.31 Mbits/sec |
※1252バイト=1280バイト(MTUサイズ)-20バイト(IPヘッダ)-8バイト(UDPヘッダ)
結果考察
パケロス・低スループットとなったケースの原因は2つあり。
確認した内容
No5,6,9,10iperf3実施中はルータAにSSH接続できなかった。
IPSEC-VPNのTunnelのMTUは1280バイトの為、それ以上のサイズのパケットを送信するとPMTUDが動作し、Ubuntuは経路毎にMTUサイズをキャッシュする。
初回のiperf3実行時はエラーになる。理由はPMTUDでICMPでMTUサイズの通知を受ける為。この時、MTUがキャッシュされる為、2回目以降は、キャッシュが残ってる間は成功する。
# ip route get 192.168.100.251
192.168.100.251 via 192.168.0.1 dev enp0s25 src 192.168.0.250 uid 0
cache expires 200sec mtu 1280
ホストAからパケットを送出した時点でフラグメントされていた。
※PMTUDのあるべき動きと異なる。本来はPMTUDにより、アプリは縮小したMTUサイズのパケットで通信する為、フラグメントされたパケットは発生しない。しかし、iperf3コマンドはサイズ指定によりMTUサイズを超過したパケットを生成し、送信元ホストでフラグメントしている。
補足
ログ
No1) UDP (-l 1252 -b 10M)
# iperf3 -c 192.168.100.251 -l 1252 -u -b 10M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1252 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.000 ms 0/29952 (0%) sender [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.017 ms 0/29952 (0%) receiver iperf Done.
No2) UDP (-l 1253 -b 10M)
# iperf3 -c 192.168.100.251 -l 1253 -u -b 10M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1253 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.000 ms 0/29928 (0%) sender [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.080 ms 0/29926 (0%) receiver iperf Done.
No3) UDP (-l 1472 -b 10M)
# iperf3 -c 192.168.100.251 -l 1472 -u -b 10M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1472 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.000 ms 0/25475 (0%) sender [ 5] 0.00-30.00 sec 35.8 MBytes 10.0 Mbits/sec 0.062 ms 0/25474 (0%) receiver iperf Done.
No4) UDP (-l 1252 -b 70M)
# iperf3 -c 192.168.100.251 -l 1252 -u -b 70M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1252 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 250 MBytes 70.0 Mbits/sec 0.000 ms 0/209659 (0%) sender [ 5] 0.00-30.00 sec 250 MBytes 70.0 Mbits/sec 0.176 ms 0/209658 (0%) receiver iperf Done.
No5) UDP (-l 1253 -b 70M)
# iperf3 -c 192.168.100.251 -l 1253 -u -b 70M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1253 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 250 MBytes 70.0 Mbits/sec 0.000 ms 0/209491 (0%) sender [ 5] 0.00-30.21 sec 39.0 MBytes 10.8 Mbits/sec 1.167 ms 176850/209483 (84%) receiver iperf Done.
No6) UDP (-l 1472 -b 70M)
# iperf3 -c 192.168.100.251 -l 1472 -u -b 70M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1472 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 250 MBytes 70.0 Mbits/sec 0.000 ms 0/178324 (0%) sender [ 5] 0.00-30.21 sec 44.8 MBytes 12.4 Mbits/sec 1.114 ms 146374/178303 (82%) receiver iperf Done.
No7) UDP (-l 1252 -b 80M)
# iperf3 -c 192.168.100.251 -l 1252 -u -b 80M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1252 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 286 MBytes 80.0 Mbits/sec 0.000 ms 0/239610 (0%) sender [ 5] 0.00-30.00 sec 286 MBytes 80.0 Mbits/sec 0.170 ms 0/239610 (0%) receiver iperf Done.
No8) UDP (-l 1252 -b 100M)
# iperf3 -c 192.168.100.251 -l 1252 -u -b 100M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1252 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 358 MBytes 100 Mbits/sec 0.000 ms 0/299512 (0%) sender [ 5] 0.00-30.01 sec 309 MBytes 86.4 Mbits/sec 0.164 ms 40609/299512 (14%) receiver iperf Done.
No9) UDP (-l 1472 -b 100M)
# iperf3 -c 192.168.100.251 -l 1472 -u -b 100M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1472 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 358 MBytes 100 Mbits/sec 0.000 ms 0/254748 (0%) sender [ 5] 0.00-30.02 sec 43.0 MBytes 12.0 Mbits/sec 1.235 ms 223833/254439 (88%) receiver iperf Done.
No10) UDP (-l 1472 -b 1000M)
# iperf3 -c 192.168.100.251 -l 1472 -u -b 1000M -t 30 | grep -B4 "iperf Done." warning: UDP block size 1472 exceeds TCP MSS 1228, may result in fragmentation / drops [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-30.00 sec 3.07 GBytes 880 Mbits/sec 0.000 ms 0/2241581 (0%) sender [ 5] 0.00-30.21 sec 15.5 MBytes 4.31 Mbits/sec 1.259 ms 845292/856348 (99%) receiver iperf Done.
ルーターAコンフィグ
login password * administrator password * login user HOGE * console character ascii login timer 300 ip route 192.168.100.0/24 gateway tunnel 1 ip lan1 address 192.168.0.1/24 ip lan2 address 192.168.200.1/24 provider lan1 name LAN: tunnel select 1 ipsec tunnel 101 ipsec sa policy 101 1 esp aes-cbc sha-hmac ipsec ike keepalive log 1 off ipsec ike keepalive use 1 on heartbeat ipsec ike local address 1 192.168.200.1 ipsec ike pre-shared-key 1 * ipsec ike remote address 1 192.168.200.2 ip tunnel tcp mss limit auto tunnel enable 1 ipsec auto refresh on dhcp service server dhcp server rfc2131 compliant except remain-silent dhcp scope 1 192.168.0.11-192.168.0.150/24 sshd service on sshd host key generate *
ルーターBコンフィグ
login password encrypted * administrator password encrypted * login user HOGE * console character ascii login timer 300 ip route 192.168.0.0/24 gateway tunnel 1 ip lan1 address 192.168.100.1/24 ip lan2 address 192.168.200.2/24 tunnel select 1 ipsec tunnel 101 ipsec sa policy 101 1 esp aes-cbc sha-hmac ipsec ike keepalive log 1 off ipsec ike keepalive use 1 on ipsec ike local address 1 192.168.200.2 ipsec ike pre-shared-key 1 * ipsec ike remote address 1 192.168.200.1 tunnel enable 1 ipsec auto refresh on dhcp service server dhcp server rfc2131 compliant except remain-silent dhcp scope 1 192.168.100.2-192.168.100.191/24 sshd service on sshd host key generate *
