TCP Hollywood at IFIP Networking 2016
Stephen McQuistin presented our paper TCP Hollywood: An Unordered, Time-Lined, TCP for Networked Multimedia Applications at the IFIP Networking 2016 conference in Vienna, Austria, on 19 May 2016 (slides). This paper describes the design of TCP Hollywood, and presents analysis showing when it will improve performance compared to standard TCP.
Ossification of the transport-layer forces networked multimedia applications to use TCP or UDP, but neither protocol is well suited to the task. UDP contributes minimal latency, making it the recommended transport to meet the strict latency bounds of real-time applications, but provides limited support to applications, and is commonly blocked by enterprise firewalls. TCP prefers reliability to timeliness, and its congestion control tends to drive up queueing delay, but is often the only transport that can pass through middleboxes on the path. Accordingly, and despite its many problems, TCP is rapidly becoming the de facto transport for multimedia traffic. To improve transport for these applications, we present TCP Hollywood, an unordered, time-lined, TCP variant designed to support real-time multimedia traffic while being widely deployable.
TCP Hollywood is wire compatible with standard TCP, but eliminates two sources of transport-induced latency, and provides reliability semantics that better suit real-time multimedia applications. Specifically, TCP Hollywood: 1) removes head-of-line blocking at the receiver and delivers received data to the application immediately, irrespective of ordering; and 2) relaxes reliability to respect time lines provided by the application, so only data that will arrive in time is retransmitted, otherwise retransmissions carry new data. The combination of both design elements reduces latency and introduces message-oriented semantics, allowing TCP Hollywood to express inter-dependencies between messages. Crucially, TCP Hollywood is wire-compatible with TCP, and incrementally deployable on the public Internet.
Head-of-line blocking occurs with TCP when packet loss occurs. Because TCP provides a reliable, ordered, byte stream service, any lost packets must be retransmitted before the following data can be delivered to the application. This can take a long time, leading to a stall in the media play-out of a real-time application while waiting for retransmission. TCP Hollywood addresses this in two ways. Firstly, by adding a framing layer, using COBS encoding implemented in an user-space intermediary layer that sits over a modified TCP stack, and provides a message-oriented delivery service. This allows complete messages received after a packet loss to be delivered to the application out of order, without waiting for retransmission.
Secondly, TCP Hollywood allows inconsistent retransmissions. That is, data that cannot be delivered before the application deadline is replaced by new data, sent in a TCP segment with the same sequence number as the lost data to complete the TCP sequence number space. We extend the Sockets API to allow applications to specify deadlines for delivery of messages, allowing the kernel to determine, based on RTT and deadline, when a message has expired and when to send new data in a retransmitted segment. We also allow dependencies between messages to be signalled, to avoid sending data that depends on a previous message that was known to be lost. Finally, because TCP Hollywood provides a message delivery service, rather than a byte stream, it's straight forward to provide demultiplexing of messages onto separate channels.
The structure of an TCP Hollywood implementation is shown in the figure below. It comprises an intermediary layer that provides the message segmentation and channel handling, plus extensions to the kernel TCP stack that implement inconsistent retransmission, expiry of messages that will not arrive in time, and dependency tracking.
The benefits of TCP Hollywood for reducing latency depend on the relative values of the network round-trip time, Trtt, the media framing interval, and the play-out delay. A real-time application will have a bound on the arrival time of data if it is to be played out in order, and with the correct timing. We denote this bound as Tmax. Taking into account this bound, the amount of media encoded into each message Tframing, and Trtt, we derive an upper bound on the play-out delay at the receiver of Tplayout ≤ Tmax − Tframing − Trtt/2 if the media is to arrive in time to be useful. We denote this as the application deadline. We also know the retransmission time for a lost segment, being equal to the sum of the round-trip time, the duration of the media encoded in a segment, and the time for the following three segments to arrive in order to generate a triple duplicate ACK. This gives a lower-bound on the play-out delay, if the retransmission is to be useful. As the figure below shows, this gives a limited region where the play-out delay is small enough that the application deadline is met, but large enough for a retransmission to arrive, and a potentially larger region where retransmissions arrive too late to be useful. It is this region of wasted TCP retransmissions that we target with inconsistent retransmissions.
We expand on this analysis in the paper. Assuming a typical 150ms delay bound for voice telephony, inconsistent retransmissions can prevent stalls on intercontinental calls running over TCP Hollywood, although they offer limited benefit for more local calling. Our NOSSDAV 2016 paper discusses other scenarios where inconsistent retransmissions can have practical benefit for video streaming.
TCP Hollywood ought to be entirely compatible with TCP. The only on-the-wire visible difference between a TCP Hollywood flow and a standard TCP flow appears within the payload data carried by inconsistent retransmissions. Recall from Section III-A that inconsistent retransmissions carry new payload data inside of segments with previously transmitted sequence numbers. This modification is invisible to receivers and middleboxes that only process TCP/IP headers, but is visible to middleboxes that use deep packet inspection if they compare the contents of a retransmitted packet with the original data. Depending on the configuration such behaviour may disrupt the connection. For example, a firewall may interpret inconsistent retransmissions as belonging to a man-on-the-side attack, and reset the connection.
We conducted experiments with a live deployment of TCP Hollywood to obtain an initial assessment on whether such middleboxes exist, and what impact they have. A TCP Hollywood server was setup on a public IP address, and configured to always send inconsistent retransmissions in lieu of the original data, so that all retransmissions contained new data with the same sequence numbers. The server was configured to listen on ports 80, 4001, and 5001. Port 80 is used by web traffic, and can be expected to be affected by middleboxes such as "transparent" caches and firewalls. We expect ports 4001 and 5001 to be less likely to be subject to interference by middleboxes, since they are not used by popular applications.
The evaluation was conducted using clients in 14 different locations in the UK, connecting to a server located at the University of Glasgow. The clients connected via eight different fixed-line residential ISPs (Andrews & Arnold, BT, Demon, EE, Eclipse, Sky, TalkTalk, and Virgin), and four mobile operators (EE, O2, Three, and Vodafone). All of the fixed-line residential ISPs successfully delivered the inconsistent retransmissions. In contrast only one out of the four mobile operators delivered inconsistent retransmissions. The three remaining mobile operators delivered the original segments instead, while the server saw no corresponding segment loss. The observed behaviour is consistent with a transparent split-connection TCP performance enhancing proxy cache that intercepts and responds to ACKs from the client on behalf of the server. On two of the three providers, this caching behaviour was seen on both port 80 and port 4001, while the other provider appeared to operate a cache on port 80 only.
Crucially, TCP Hollywood continued to operate whether or not the middlebox was present in the network. At no time did connections suffer a reset, and the use of the TCP Hollywood extensions did not affect connectivity or performance. Our sample size is small, but this experiment provides evidence that TCP Hollywood continues to deliver messages and eliminate head-of-line blocking, even when inconsistent retransmissions are absent. In the worst-case, performance is the same as TCP without our extensions.