Simulated Network

There are may ways in which an International NZ/US connection might be deployed as part of an ISP's network. This paper considers two architectures.

US and NZ proxies

The main elements of the first of these architectures is shown in figure 1. The diagram shows web clients located in NZ connecting to a NZ proxy. This proxy in turn connects to a US based proxy which, in its turn, connects to web servers based in the US. There are three TCP connections involved in fetching a web page. The first connects the web client to the NZ proxy, the second is between the two proxies and the third is from the US proxy to the US server. Multiplexing is implemented between the proxies. That is the data for different replies may be interleaved on a single TCP connection between the proxies. The overhead of multiplexing is assumed to be, on average, 20 bytes per HTTP reply segment received from an HTTP server. It is expected that multiplexing will improve the efficiency of the international link. Because TCP does not maintain the boundaries between application requests the data from (possibly different) HTTP reply packets may be repackaged for more efficient TCP transmission. In most cases the TCP segments will be the maximum MSS size. Because the connections between the proxies persist indefinitely the effect of TCP slow start is greatly reduced over the international component of the network, which is where slow start would otherwise have the greatest effect. The number of concurrent TCP connections between the proxies may be limited. Each active TCP connection consumes resources and this imposes a maximum value. This parameter may als be tuned to improve performance. if a very large number of concurrent connections are available some of the performance advantages described previously will not occur. For example the effect of slow start will be felt by a larger proportion of HTTP requests as the number of TCP connections between the proxies increases. We investigate this issue by simulating differeing number of connections between the proxies.

NZ only proxy

A second, simpler, case is also considered. In this second case the US proxy is omitted. Only two TCP connections are involved with fetching a web page. The first, from the web client to the NZ proxy, is the same as in the US-proxy case. The second connection is from the NZ proxy to the US server providing the web page. In this case there is a TCP connection across the international link for each HTTP request.

Table 1: Main Network Parameters

US/NZ Bandwidth	34.368Mbps (E3)
US/NZ Delay	320ms[3]
TCP buffer size
Proxies	32767
Servers	as measured
Maximum Segment Size
Between proxies	1460
Elsewhere	as measured
Delays in US cloud	as measured
Delays in NZ cloud	not simulated

The main parameters of the network are shown in table 1. The values have been chosen to match real network parameters where possible. As noted above the bandwith delay product of the network has an impact on TCP performance. The bandwidth delay product for this network is given by:

$BDP = D * 2 * B$

where:	$DELAY$	is the trans-pacific latency
	$B$	is the link bandwidth

so

$\begin{tabular}{ll} $BDP$\ &= $( 0.060 * 2 ) * 34.368$\\ &= $ 4.1$\ megabits\\ \end{tabular}$

This means that the TCP retransmit buffers need to be larger than 516kb to allow this link to be filled by a single TCP connection. If smaller buffers are used the buffer will be filled before the first data sent has been acknowledged and the flow of data into the link will be suspended while the transmitter waits for an acknowledgment. Standard TCP limits the window size to 64kb[4]. The big window extension for RFC1323[5] extends this limit to $2^{32}$. Use of this option is not widespread and is outside the control of the ISP. If an implementation is limited to 32767 bytes (a common implementation maximum) then the maximum bit rate for a single TCP connection over this link is:

$\begin{tabular}{ll} $MBR$\ &= $S / ( D * 2 )$\\ &= $32767 * 8 / ( 0.060 * 2 )$\\ &= $2.2 Mbps$\\ \end{tabular}$

where:	$D$	is the trans-pacific latency
	$S$	is the buffer size

This effect is seen in simulation of the link and is shown in figure 4. This graph shows that the bandwidth consumed by a single TCP connection as the offered load increases plateaus at around 2.2Mbps. This bandwidth limitation is expected to impact negatively on the performance of the international connection, especially where a US based proxy is used and the number of connections between the US and NZ proxies is small. Real International architectures would be not be as simple as the one described in this paper. Most will need more than a single proxy at each end of the international link to support the required load. The NZ proxy would almost certainly include a cache that satisfies some of the HTTP requests locally. There are many routers not shown. The simpler architecture used in this paper makes the simulation easier and shows the differences between the international implementations without interference from the full range of factors that would impact the performance of a real system.