EC2 & ELB Networking SockPerf TEST (Zone/Public/Private)

• EC2 and ELB Networking Latency TEST
  • 제반 사항
  • VM 구성
    • 구성도
    • 설정 내역
  • 결과
    • 동일 Zone
    • 다른 Zone

제반 사항

(Note)
- 부하(Load)나, 외부상황은 배제한, EC2 Instance간의 Network Latency 테스트.
- 횟수가 많아야 신뢰도가 있는 자료가 나오겠으나, 시간상 1회/10초 수행.
- "Percentile 99.99" 와 "Average" 그래프화. 

EC2-Latency.xlsx

• Region: Tokyo
• 인스턴스(VM): 3EA
• 인스턴스 OS: Ubuntu Server 12.04.2 LTS (64bit)
• 인스턴스 스펙: M1 Small (m1.small, 1.7 GiB)
• 테스트 상세
  • 도구: sockperf_2.5.193 (https://code.google.com/p/sockperf/downloads/list)
  • 모드: ping-pong (run sockperf client for latency test in ping pong mode)
  • 방식: RTT (Round Trip Time)
  • 시간: 10sec
  • Size: 16KB (16384 Byte)
  • 단위: usec (마이크로초)
  • Command
    • Server: ./sockperf server -p 5001 {--tcp}
    • Client: ./sockperf ping-pong -i {Server-IP} -p 5001 -m 16384 -t 10 --pps=max {--tcp}
예상 Case
• 동일 Zone에 존재하는 VM to VM
  • 공인 DNS (TCP/UDP)
  • 사설 DNS (TCP/UDP)
  • ELB 경유 (TCP/UDP)
다른 Zone에 존재하는 VM to VM
• 공인 DNS (TCP/UDP)
• 사설 DNS (TCP/UDP)
• ELB 경유 (TCP/UDP)

VM 구성

구성도

설정 내역

• VM-A
  • Zone: ap-northeast-1a
  • Privat DNS: 10.152.173.96 (ip-10-152-173-96.ap-northeast-1.compute.internal)
  • Public DNS: 54.249.200.29 (ec2-54-249-200-29.ap-northeast-1.compute.amazonaws.com)
• VM-B
  • Zone: ap-northeast-1a
  • Privat DNS: 10.162.15.46 (ip-10-162-15-46.ap-northeast-1.compute.internal)
  • Public DNS: 54.250.97.46 (ec2-54-250-97-46.ap-northeast-1.compute.amazonaws.com)
• VM-C
  • Zone: ap-northeast-1c
  • Privat DNS: 10.121.6.240 (ip-10-121-6-240.ap-northeast-1.compute.internal)
  • Public DNS: 54.250.69.124 (ec2-54-250-69-124.ap-northeast-1.compute.amazonaws.com)

• ELB-B 정보
  • Zone: ap-northeast-1a
  • LB Protocol: TCP
  • LB Port: 5001
  • Instance Protocol: TCP
  • Instance Port: 5001
  • DNS/IP
    • 176.32.93.150 / SockPerf-1694303445.ap-northeast-1.elb.amazonaws.com (A Record)

• ELB-C 정보
  • Zone: ap-northeast-1c
  • LB Protocol: TCP
  • LB Port: 5001
  • Instance Protocol: TCP
  • Instance Port: 5001
  • DNS/IP
    • 54.248.124.224 / sockperf-c-1386190870.ap-northeast-1.elb.amazonaws.com (A Record)

결과

첨부 엑셀/그래프 참조

동일 Zone

A->B: 공인 DNS
• TCP
  

  root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 54.250.97.46 -p 5001 -m 16384 -t 10 --pps=max --tcp
  sockperf:  == version #2.5. == 
  sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
  
  [ 0] IP = 54.250.97.46    PORT =  5001 # TCP
  sockperf: Warmup stage (sending a few dummy messages)...
  sockperf: Starting test...
  sockperf: Test end (interrupted by timer)
  sockperf: Test ended
  sockperf: [Total Run] RunTime=10.099 sec; SentMessages=3455; ReceivedMessages=3454
  sockperf: ========= Printing statistics for Server No: 0
  sockperf: [Valid Duration] RunTime=9.998 sec; SentMessages=3417; ReceivedMessages=3417
  sockperf: ====> avg-lat=1462.875 (std-dev=1090.760)
  sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
  sockperf: Summary: Latency is 1462.875 usec
  sockperf: Total 3417 observations; each percentile contains 34.17 observations
  sockperf: ---> <MAX> observation = 31958.137
  sockperf: ---> percentile  99.99 = 31958.137
  sockperf: ---> percentile  99.90 = 18919.201
  sockperf: ---> percentile  99.50 = 3809.342
  sockperf: ---> percentile  99.00 = 3328.685
  sockperf: ---> percentile  95.00 = 2655.743
  sockperf: ---> percentile  90.00 = 2176.901
  sockperf: ---> percentile  75.00 = 1718.117
  sockperf: ---> percentile  50.00 = 1237.302
  sockperf: ---> percentile  25.00 = 1022.388
  sockperf: ---> <MIN> observation =  721.464

• UDP
  

  root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 54.250.97.46 -p 5001 -m 16384 -t 10 --pps=max
  sockperf:  == version #2.5. == 
  sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
  
  [ 0] IP = 54.250.97.46    PORT =  5001 # UDP
  sockperf: Warmup stage (sending a few dummy messages)...
  sockperf: Starting test...
  sockperf: Test end (interrupted by timer)
  sockperf: Test ended
  sockperf: [Total Run] RunTime=10.101 sec; SentMessages=1172; ReceivedMessages=1171
  sockperf: ========= Printing statistics for Server No: 0
  sockperf: [Valid Duration] RunTime=9.997 sec; SentMessages=1159; ReceivedMessages=1159
  sockperf: ====> avg-lat=4312.799 (std-dev=1570.727)
  sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
  sockperf: Summary: Latency is 4312.799 usec
  sockperf: Total 1159 observations; each percentile contains 11.59 observations
  sockperf: ---> <MAX> observation = 34023.287
  sockperf: ---> percentile  99.99 = 34023.287
  sockperf: ---> percentile  99.90 = 32017.070
  sockperf: ---> percentile  99.50 = 8009.275
  sockperf: ---> percentile  99.00 = 8002.157
  sockperf: ---> percentile  95.00 = 6002.642
  sockperf: ---> percentile  90.00 = 5995.283
  sockperf: ---> percentile  75.00 = 4005.037
  sockperf: ---> percentile  50.00 = 4000.878
  sockperf: ---> percentile  25.00 = 3997.844
  sockperf: ---> <MIN> observation = 3441.178
  

• A->B: 사설 DNS
  • TCP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 10.162.15.46 -p 5001 -m 16384 -t 10 --pps=max --tcp
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 10.162.15.46    PORT =  5001 # TCP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.101 sec; SentMessages=3850; ReceivedMessages=3849
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=10.001 sec; SentMessages=3814; ReceivedMessages=3814
    sockperf: ====> avg-lat=1310.974 (std-dev=1300.552)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 1310.974 usec
    sockperf: Total 3814 observations; each percentile contains 38.14 observations
    sockperf: ---> <MAX> observation = 31679.585
    sockperf: ---> percentile  99.99 = 31679.585
    sockperf: ---> percentile  99.90 = 18777.097
    sockperf: ---> percentile  99.50 = 5004.997
    sockperf: ---> percentile  99.00 = 4177.719
    sockperf: ---> percentile  95.00 = 2647.471
    sockperf: ---> percentile  90.00 = 2222.535
    sockperf: ---> percentile  75.00 = 1618.295
    sockperf: ---> percentile  50.00 = 1085.067
    sockperf: ---> percentile  25.00 =  714.380
    sockperf: ---> <MIN> observation =  347.386
    

  • UDP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 10.162.15.46 -p 5001 -m 16384 -t 10 --pps=max
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 10.162.15.46    PORT =  5001 # UDP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.101 sec; SentMessages=1231; ReceivedMessages=1230
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=9.601 sec; SentMessages=1224; ReceivedMessages=1224
    sockperf: ====> avg-lat=3921.816 (std-dev=1746.140)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 3921.816 usec
    sockperf: Total 1224 observations; each percentile contains 12.24 observations
    sockperf: ---> <MAX> observation = 34020.301
    sockperf: ---> percentile  99.99 = 34020.301
    sockperf: ---> percentile  99.90 = 32013.095
    sockperf: ---> percentile  99.50 = 8008.463
    sockperf: ---> percentile  99.00 = 7998.229
    sockperf: ---> percentile  95.00 = 5999.880
    sockperf: ---> percentile  90.00 = 4990.765
    sockperf: ---> percentile  75.00 = 4005.645
    sockperf: ---> percentile  50.00 = 3999.775
    sockperf: ---> percentile  25.00 = 3990.058
    sockperf: ---> <MIN> observation = 1941.773
    

• A->B: ELB-B 경유
  • TCP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 176.32.93.150 -p 5001 -m 16384 -t 10 --pps=max --tcp
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 176.32.93.150   PORT =  5001 # TCP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.100 sec; SentMessages=2628; ReceivedMessages=2627
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=9.955 sec; SentMessages=2605; ReceivedMessages=2605
    sockperf: ====> avg-lat=1910.550 (std-dev=948.343)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 1910.550 usec
    sockperf: Total 2605 observations; each percentile contains 26.05 observations
    sockperf: ---> <MAX> observation = 31351.928
    sockperf: ---> percentile  99.99 = 31351.928
    sockperf: ---> percentile  99.90 = 13266.080
    sockperf: ---> percentile  99.50 = 4643.678
    sockperf: ---> percentile  99.00 = 4265.762
    sockperf: ---> percentile  95.00 = 3234.758
    sockperf: ---> percentile  90.00 = 2781.928
    sockperf: ---> percentile  75.00 = 2069.907
    sockperf: ---> percentile  50.00 = 1670.907
    sockperf: ---> percentile  25.00 = 1427.325
    sockperf: ---> <MIN> observation = 1145.673
    

  • UDP
    • ELB에서 UDP 미지원으로 테스트 불가

다른 Zone

• A->C: 공인 DNS
  • TCP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 54.250.69.124 -p 5001 -m 16384 -t 10 --pps=max --tcp
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 54.250.69.124   PORT =  5001 # TCP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.102 sec; SentMessages=1982; ReceivedMessages=1981
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=10.001 sec; SentMessages=1962; ReceivedMessages=1962
    sockperf: ====> avg-lat=2548.662 (std-dev=1832.228)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 2548.662 usec
    sockperf: Total 1962 observations; each percentile contains 19.62 observations
    sockperf: ---> <MAX> observation = 46399.863
    sockperf: ---> percentile  99.99 = 46399.863
    sockperf: ---> percentile  99.90 = 26070.326
    sockperf: ---> percentile  99.50 = 12983.683
    sockperf: ---> percentile  99.00 = 4294.068
    sockperf: ---> percentile  95.00 = 4025.450
    sockperf: ---> percentile  90.00 = 3924.323
    sockperf: ---> percentile  75.00 = 2629.824
    sockperf: ---> percentile  50.00 = 2070.109
    sockperf: ---> percentile  25.00 = 1963.635
    sockperf: ---> <MIN> observation = 1606.874
    

  • UDP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 54.250.69.124 -p 5001 -m 16384 -t 10 --pps=max
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 54.250.69.124   PORT =  5001 # UDP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.101 sec; SentMessages=1153; ReceivedMessages=1152
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=10.001 sec; SentMessages=1140; ReceivedMessages=1140
    sockperf: ====> avg-lat=4386.197 (std-dev=1533.904)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 4386.197 usec
    sockperf: Total 1140 observations; each percentile contains 11.40 observations
    sockperf: ---> <MAX> observation = 28006.313
    sockperf: ---> percentile  99.99 = 28006.313
    sockperf: ---> percentile  99.90 = 26138.757
    sockperf: ---> percentile  99.50 = 12003.960
    sockperf: ---> percentile  99.00 = 10006.633
    sockperf: ---> percentile  95.00 = 6004.485
    sockperf: ---> percentile  90.00 = 5995.108
    sockperf: ---> percentile  75.00 = 4007.216
    sockperf: ---> percentile  50.00 = 4001.184
    sockperf: ---> percentile  25.00 = 3996.925
    sockperf: ---> <MIN> observation = 3253.957
    

• A->C: 사설 DNS
  • TCP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 10.121.6.240 -p 5001 -m 16384 -t 10 --pps=max --tcp
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 10.121.6.240    PORT =  5001 # TCP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.100 sec; SentMessages=2316; ReceivedMessages=2315
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=10.003 sec; SentMessages=2297; ReceivedMessages=2297
    sockperf: ====> avg-lat=2177.273 (std-dev=1583.329)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 2177.273 usec
    sockperf: Total 2297 observations; each percentile contains 22.97 observations
    sockperf: ---> <MAX> observation = 26539.603
    sockperf: ---> percentile  99.99 = 26539.603
    sockperf: ---> percentile  99.90 = 24238.810
    sockperf: ---> percentile  99.50 = 12957.637
    sockperf: ---> percentile  99.00 = 11637.611
    sockperf: ---> percentile  95.00 = 2913.860
    sockperf: ---> percentile  90.00 = 2579.413
    sockperf: ---> percentile  75.00 = 2088.335
    sockperf: ---> percentile  50.00 = 1997.917
    sockperf: ---> percentile  25.00 = 1768.126
    sockperf: ---> <MIN> observation = 1274.989
    

  • UDP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 10.121.6.240 -p 5001 -m 16384 -t 10 --pps=max
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 10.121.6.240    PORT =  5001 # UDP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.101 sec; SentMessages=934; ReceivedMessages=933
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=9.997 sec; SentMessages=924; ReceivedMessages=924
    sockperf: ====> avg-lat=5409.573 (std-dev=1625.872)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 5409.573 usec
    sockperf: Total 924 observations; each percentile contains 9.24 observations
    sockperf: ---> <MAX> observation = 25542.427
    sockperf: ---> percentile  99.99 = 25542.427
    sockperf: ---> percentile  99.90 = 20010.379
    sockperf: ---> percentile  99.50 = 12113.971
    sockperf: ---> percentile  99.00 = 8020.686
    sockperf: ---> percentile  95.00 = 7988.814
    sockperf: ---> percentile  90.00 = 6012.314
    sockperf: ---> percentile  75.00 = 6003.931
    sockperf: ---> percentile  50.00 = 5994.757
    sockperf: ---> percentile  25.00 = 4001.799
    sockperf: ---> <MIN> observation = 1996.650
    

• A->C: ELB-C 경유
  • TCP
    

    root@ip-10-152-173-96:/usr/local/sockperf/bin# ./sockperf ping-pong -i 54.248.124.224 -p 5001 -m 16384 -t 10 --pps=max --tcp
    sockperf:  == version #2.5. == 
    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
    
    [ 0] IP = 54.248.124.224  PORT =  5001 # TCP
    sockperf: Warmup stage (sending a few dummy messages)...
    sockperf: Starting test...
    sockperf: Test end (interrupted by timer)
    sockperf: Test ended
    sockperf: [Total Run] RunTime=10.099 sec; SentMessages=1579; ReceivedMessages=1578
    sockperf: ========= Printing statistics for Server No: 0
    sockperf: [Valid Duration] RunTime=10.000 sec; SentMessages=1562; ReceivedMessages=1562
    sockperf: ====> avg-lat=3200.865 (std-dev=1955.397)
    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
    sockperf: Summary: Latency is 3200.865 usec
    sockperf: Total 1562 observations; each percentile contains 15.62 observations
    sockperf: ---> <MAX> observation = 25292.453
    sockperf: ---> percentile  99.99 = 25292.453
    sockperf: ---> percentile  99.90 = 24183.071
    sockperf: ---> percentile  99.50 = 19981.548
    sockperf: ---> percentile  99.00 = 14068.981
    sockperf: ---> percentile  95.00 = 4114.684
    sockperf: ---> percentile  90.00 = 4015.404
    sockperf: ---> percentile  75.00 = 3850.997
    sockperf: ---> percentile  50.00 = 2928.466
    sockperf: ---> percentile  25.00 = 2218.596
    sockperf: ---> <MIN> observation = 1889.028
    

  • UDP
    • ELB에서 UDP 미지원으로 테스트 불가

(그외 유사 구글 검색 자료): http://orensol.com/2009/05/24/network-latency-inside-and-across-amazon-ec2-availability-zones/