Lecture 5 discusses reliable and unreliable data transfer in the
Internet. It explains the best-effort nature of packet delivery,
the end-to-end argument, and the timeliness-vs-reliability trade-off
inherent in the design of the Internet. And it discusses three
transport protocols in use in the Internet, UDP, TCP, and QUIC,
and how the provide different degrees of timeliness and reliability,
and offer different services to applications.
The first part of the lecture discusses packet loss in the Internet.
It talks about the causes of packet loss, the end-to-end argument,
and the timeliness-reliability trade-off.
Slides for part 1
In this lecture I want to move
on from the discussion of connection establishment,
and talk instead about reliability and effective
data transfer across the network.
There are four parts to this.
In this first part, I’ll talk briefly
about packet loss in the Internet,
and the trade-off between reliability and timeliness.
Then, I’ll move on to discuss unreliable
data using UDP, and talk about the
types of applications that benefit from this.
In part three, I’ll talk about reliable
data transfer with TCP. I’ll discuss the
TCP service model, how TCP ensures data
is delivered reliably, and some of the
limitations of TCP relating to head-of-line blocking.
Then, in the final part, I’ll conclude
by discussing how QUIC transfers data and
how this differs from TCP.
I want to start by discussing packet loss in the Internet.
What we mean when we say that the Internet
provides a best effort service.
The end-to-end argument.
And the timeliness vs reliability trade-off inherent
in the design of the Internet.
As we discussed back in lecture 1,
the Internet is a best effort packet delivery network.
This means that it’s unreliable by design.
IP packets can be lost, delayed,
reordered, or corrupted in transit. And this
is regarded as a feature, rather than a bug.
A network that can’t deliver
a packet is supposed to discard it.
There are many reasons why a packet
can get lost or discarded. It could
be due to a transmission error,
where electrical noise of wireless interference corrupts
the packet in transit, making the packet unreadable.
Or it could be because too much
traffic is arriving at some intermediate link
in the network, so an intermediate router
runs out of buffer space. If traffic
is arriving at a router from several
different incoming links, but all going to
the same destination, so it’s arriving faster
than it can be delivered, a queue
of packets will build up, waiting for transmission.
If this situation persists, the queue might
grow so much that a router runs
out of memory, and has no choice
but to discard the packets.
Or packets could be lost because of
a link failure. Or a router bug.
Or for other reasons.
How often this happens varies significantly.
The packet loss rate depends on the type of link.
Wireless links tend to be less reliable
than wired links, for example.
It’s reasonably likely that packet sent over a wireless
link, such as WiFi or 4G,
will be corrupted in transit due to
noise, interference, or cross traffic.
This is very unlikely on an Ethernet
or optical fibre link.
The packet loss rate also depends on
the overall quality and robustness of the infrastructure.
Countries with well developed
and well maintained infrastructure
tend to have reliable Internet links;
countries with less robust or lower
capacity infrastructure tend to see more problems.
And the loss rate depends on the protocol.
Some protocols intentionally try to push
links to capacity, causing temporary overload as
they try to find the limit,
as they try to find the maximum
transmission rate they can achieve.
TCP and QUIC do this in many cases,
depending on the congestion control algorithm
used, as we’ll see in lecture 6.
Other applications, such as telephony or video
conferencing, tend to have an upper bound
in the amount of data they can send.
Whatever the reason, though,
some packet loss is inevitable.
The transport layer needs to recognise this.
It must detect packet loss. And,
if the application needs reliability, it must
retransmit or otherwise repair any lost data.
That the Internet provides best effort packet
delivery is a result of the end-to-end argument.
The end-to-end argument considers whether it’s better
to place functionality inside the network or
at the end points.
For example, rather than provide best effort
delivery, we could try to make the
network deliver packets reliably. We could design
some way to detect packet loss on
a particular link, and request that the
lost packets be retransmitted locally,
somewhere within the network.
And, indeed, some network links do this.
In WiFi networks, for example, the base
station acknowledges packets it receives from the
clients, and requests any corrupted packets are
re-sent, to correct the error.
The problem is, that unless this mechanism
is 100% perfect all the time,
then end systems will still need to
check if the data has been received
correctly, and will still need some way
of retransmitting packets in the case of problems.
And if they’ve got that, why bother
with the in-network retransmission and repair?
Often times, if you add features into
the network routers, they end up duplicating
functionality that the network endpoints need to
Maybe the performance benefit of adding features
to the network is so big that it’s worth while.
But often, the right thing to do
is to keep the network simple.
Omit anything that can be done by the endpoints.
And favour simplicity over the
absolute optimal performance.
The end-to-end argument is one of the
defining principles of the Internet. And I
think it’s still a good approach to
take, when possible. Keep the network simple, if you can
The paper linked from the slide talks
about this subject in a lot more detail.
Irrespective of whether retransmission of lost packets
happen between the endpoints or within the
network, it takes time.
This leads to a fundamental trade-off in
the design of the network.
If a connection is to be reliable,
it cannot guarantee timeliness.
It’s not possible to build absolutely perfect
network links, that never discard or corrupt
packets. There’s always some risk that the
data is lost and needs to be
retransmitted. And retransmitting a packet will always
take time, and so disrupt the timeliness of the delivery.
And similarly, if a connection is to
be timely, it cannot guarantee reliability.
There’s a trade-off to be made.
Protocols like UDP are timely but don’t
attempt to be reliable. They send packets,
and if they get lost, they get lost.
TCP and QUIC, on the other hand,
aim to be reliable. They send the
packets, and if they get lost,
they retransmit them.
And if the retransmission gets lost? They
try again, until the data eventually arrives.
As we’ll see in part 3 of
this lecture, this causes head of line
blocking, making the protocol less timely.
And other protocols, such as the Real-time
Transport Protocol, RTP, that I’ll talk about
in lecture 7, or the partially reliable
version of the Stream Control Transport Protocol,
SCTP, aim for a middle ground.
They try to correct some, but not
all, of the transmission errors. The try
to achieve a balance, a middle-ground,
between timeliness and reliability.
The different protocols exist because different applications
make different trade-offs.
Some applications prefer timeliness,
some prefer reliability.
For applications like web browsing, email,
or messaging, you want to receive all
the data. If I’m loading a web
site, I’d like it to load quickly,
sure. But I prefer for it to
load slowly, and be uncorrupted, rather than
load quickly with some parts missing.
For a video conferencing tool, like Zoom,
though, the trade-off is different. If I’m
having a conversation with someone, it’s more
important that the latency is low,
than the picture quality is perfect.
The same may be true for gaming.
And this has implications for the way
we design the network.
It means that the IP layer needs
to be unreliable. It needs to be
a best effort network.
If the IP layer is unreliable,
protocols like TCP and QUIC can sit
on top and retransmit packets to make
it reliable. A transport protocol can make
an unreliable network into a reliable one.
But if the IP layer is reliable,
if the IP layer retransmits packets itself,
then the network, the applications, the transport
protocols, can’t undo that.
So this concludes the discussion of packet
loss and why the Internet opts to
provide an unreliable, best-effort, service.
In the next part, I’ll talk about
UDP and how to make use of
an unreliable transport protocol.
The second part of the lecture discusses UDP. It outlines the UDP
service model, and reviews how to send and receive data using UDP
sockets, and the implications of unreliable delivery for applications
using UDP. It discusses how UDP is suitable for real-time applications
that prioritise low-latency over reliability. And is discusses the use
of UDP as a substrate on which alternative transport protocols can be
implemented, avoiding some of the challenges of protocol ossification.
Slides for part 2
In this part, I’ll move on to
discuss how to send unreliable data using UDP.
I’ll talk about the UDP service model,
how to send and receive packets,
and how to layer protocols on top of UDP.
UDP provides an unreliable,
connectionless, datagram service.
It adds only two features on top
of the IP layer: port numbers and a checksum.
The checksum is used to detect whether
the packet has been corrupted in transit.
If so, the packet will be discarded
by the UDP code in the operating
system of the receiver, and won’t be
delivered to the application.
The port numbers determine what application receives
the UDP datagrams when they arrive at
the destination. They’re set by the bind()
call, once the socket has been created.
The Internet Assigned Numbers Authority, the IANA,
maintains a list of well-known UDP port
numbers which you should use for particular
applications. This is linked from the bottom of the slide.
UDP is very minimal. It doesn’t provide
reliability, or ordering, or congestion control.
It just delivers packets to an application,
that’s bound to a particular port.
Mostly, UDP is used as a substrate.
It’s a base on which higher-layer protocols are built.
QUIC is an example of this,
as we discussed in the last lecture.
Others are the Real-time Transport Protocol,
and the DNS protocol,
that we’ll talk about later in the course.
UDP is connectionless. It’s got no notion
of clients or servers, or of establishing
a connection before it can be used.
To use UDP, you first create a socket.
Then you call bind(),
to choose the local port on which that socket
listens for incoming datagrams.
They you call recvfrom() if you want
to receive a datagram on that socket,
or sendto() if you want to send a datagram.
You don’t need to connect.
You don’t need to accept connections.
You just send and receive data.
And maybe that data is delivered.
When you’re finished, you close the socket.
Protocols that run on top of UDP,
such as QUIC, might add support for
connections, reliability, ordering,
congestion control, and so on,
but UDP itself supports none of this.
To send a UDP datagram, you use
the sendto() function.
This work similarly to the send() function
you used to send data over a
TCP connection in the labs, except that
it takes two additional parameters to indicate
the address to which the datagram should
be sent, and the size of that address.
When using TCP, you establish a connection
between a socket, bound to a local
address and port, and a server listening
on a particular port on some remote
IP address. And once the connection is
established, all the data goes over that
connection, to the same destination.
UDP is not like that.
Every time you call sendto(), you specify
the destination address. Every packet you send
from a UDP socket can go to
a different destination, if you want.
There’s no notion of connections.
Now, you can call connect() on a
UDP socket, if you like, but it doesn’t actually create
a connection. Rather, it just remembers the
address you give it, so you can
call send(), rather than sendto() in future,
to save having to specify the address each time.
To receive a UDP datagram, you call
the recvfrom() function, as shown on the slide.
This is like the recv() call you
use with TCP, but again it has
two additional parameters. These allow it to
record the address that the received datagram
came from, so you can use them
in the sendto() function to send a reply.
You can also call recv(), rather than
recvfrom(), like with TCP, and it works,
but it doesn’t give you the return
address, so it’s not very useful.
The important point with UDP is that
packets can be lost, delayed, or reordered
in transit, and UDP doesn’t attempt to
recover from this.
Just because you send a datagram,
doesn’t mean it will arrive. And if
datagrams do arrive, they won’t necessarily arrive
in the order sent.
Unlike TCP, where data written to a
connection in a single send() call might
end up being split across multiple read()
calls at the receiver, a single UDP
send generates exactly one datagram.
If it’s delivered at all, the data
sent by a single call to sendto()
will be delivered by a single call
to recvfrom(). UDP doesn’t split messages.
But UDP is otherwise unreliable.
Datagrams can be lost, delayed, reordered,
or duplicated in transit.
Data sent with sendto() might never arrive.
Or it might arrive more than once.
Or data sent in consecutive calls to
sendto() might arrive out of order,
with data sent later arriving first.
UDP doesn’t attempt to correct any of these things.
The protocol you build on top of
UDP might choose to do so.
For example, we saw that QUIC adds
packet sequence numbers and acknowledgement frames to
the data it sends within UDP packets.
This lets it put the data back
into the correct order, and retransmit any
But there’s no requirement that the protocol
running over UDP is reliable.
RTP, the Real-time Transport Protocol, that’s used
for video conferencing apps, puts sequence numbers
and timestamps inside the UDP datagrams it
sends, so it can know if any
data is missing, and it can conceal
loss or reconstruct the packet playout time,
but it generally doesn’t retransmit missing data.
UDP gives the application the choice of
building reliability, if it wants it.
But it doesn’t require that the applications
deliver data reliably.
Applications that use UDP need to organise
the data they send, so it’s useful
if some data is lost.
Different applications do this in different ways,
depending on their needs.
QUIC, for example, organises the data into
sub-streams within a connection,
and retransmits missing data.
Video conferencing applications
tend to do something different.
The way video compression works, is that
the codec sends occasional full frames of
video, known as I-frames, index frames,
every few seconds. And in between these
it sends only the differences from the
previous frame, known as P-frames, predicted frames.
In a video call, it’s common for
the background to stay the same,
while the person moves in the foreground,
so a lot of the frame is
the same each time. By only sending
the differences, video compression saves bandwidth.
But this affects how the application treats
the different datagrams.
If a UDP datagram containing a predicted
frame is lost, it’s not that important.
You’ll get a glitch in one frame of video.
But if a UDP datagram containing an
index frame, or part of an index
frame, is lost, then that matters a
lot more because the next few seconds
worth of video are predicted based on
that index frame. Losing an index frame
corrupts several seconds worth of video.
For this reason, many video conferencing apps
running over UDP try to determine if
missing packets contained an index frame or
not. And they try to retransmit index
frames, but not predicted frames.
The details of how they do this
aren’t really important, unless you’re building a
video conferencing app.
What’s important though, is that UDP gives
the application flexibility to be unreliable for
some of the datagrams it sends,
while trying to deliver other datagrams reliably.
You don’t have that flexibility with TCP.
UDP is harder to use, because it
provides very few services to help your
application, but it’s more flexible because you
can build exactly the services you need
on top of UDP.
Fundamentally, UDP doesn’t make any attempt to
provide sequencing, reliability,
timing recovery, or congestion control.
It just delivers datagrams on a best effort basis.
It lets you build any type of
transport protocol you want, running inside UDP packets.
Maybe that transport protocols has sequence numbers
and acknowledgements, and retransmits some or all
of the lost packets.
Maybe, instead, it uses error correcting codes,
to allow some of the packets to
be repaired without retransmission.
Maybe it includes timestamps, so the receiver
can carefully reconstruct the timing.
Maybe it contains other information.
The point is that UDP gives you
flexibility, but at the cost of having
to implement these features yourself. At the
cost of adding complexity.
There’s a lot to think about when
writing a UDP-based protocol or a UDP-
If you use a transport protocol,
like QUIC or like RTP, that runs
over UDP, then the designers of that
protocol have made these decisions, and will
have given you a library you can use.
If not, if you’re designing your own
protocol that runs over UDP, then the
IETF has written some guidelines, highlighting the
issues you need to think about,
in RFC 8085.
Please read this before you try and
write applications that use UDP. There are
a lot of non-obvious things that can catch you out.
So, that concludes our discussion of UDP.
In the next part, I’ll talk about
how TCP delivers data reliably.
The third part of the lecture discusses TCP. It outlines the TCP
service model and shows how to send and receive data using a TCP
connection. It explains how TCP ensures reliable and order data
transfer, using sequence numbers and acknowledgements. And it
explains TCP loss detection using timeouts and triple-duplicate
acknowledgements. The issue of head-of-line blocking in TCP
connections is discussed, as an example of the timeliness vs
Slides for part 3
In this part I want to talk
about how reliable data is delivered using
TCP connections. I’ll talk about the TCP
service model, how TCP uses sequence numbers
and acknowledgments, and how packet loss detection
and recovery works in TCP.
Thinking about the TCP service model,
as we've seen in previous lectures,
TCP provides a reliable, ordered, byte stream
delivery service that runs over IP.
The applications write data into the TCP
socket, that buffers it up in the
sending system, and then delivers it over
a sequence of data segments over the IP layer.
When these data packets, these data segments,
are received, they are accumulated in a
receive buffer at the receiver. If anything
is lost, or arrives out of order,
it's re-transmitted, and eventually the data is
delivered to the application.
The data delivered to the application is
always delivered reliably, and in the order sent.
If something is lost, if something needs
to be re-transmitted, this stalls the delivery
of the later data, to make sure
that everything is always delivered in order.
TCP delivers, as we say, an order,
reliable, byte stream.
After the connection has been established,
after the SYN, SYN-ACK, ACK handshake,
the client and the server can send
and receive data.
The data can flow in either direction
within that TCP connection.
It’s usual that the data follows a
request response pattern. You open the connection.
The client sends a request to the
server. The server replies with a response.
The client makes another request. The server
replies with another response, and so on.
But TCP doesn't make any requirements on
this. There’s no requirement that the data
flows in a request response pattern,
and the client and the server can
send data in any order they feel like.
TCP does ensure that the data is
delivered reliably, and in the order it
was sent, though.
TCP sends acknowledgments for each data segment
as it's received. And if any data
is lost, it retransmits that lost data.
And if segments are delayed and arrive
out of order, or if a segment
has to be re-transmitted and arrives out
of order, then TCP will reconstruct the
order before giving the segments back to the application.
In order to send data over a
TCP connection you use the send() function.
This transmits a block of data over
the TCP connection. The parameters are the
file descriptor representing the socket – the
TCP socket, the data, the length of
the data, and a flag. And the
flag field is usually zero.
The send() function blocks until all the
data can be written.
And it might take a significant amount
of time to do this, depending on
the available capacity of the network.
It also might not be able to
send all the data.
If the connection is congested, and can't
accept any more data, then the send()
function will return to indicate that it
wasn't able to successfully send all the
data that was requested.
The return value from the send() function
is the amount of data it actually
managed to send on the connection.
And that can be less than the
amount it was asked to send.
In which case, you need to figure
out what data was not sent,
by looking at the return value,
and the amount you asked for,
and re-send just the missing part in another call.
Similarly, if an error occurs, if the
connection has failed for some reason,
the send() function will return -1,
and it will set the global variable
errno to indicate that.
On the receiving side you call the
recv() function to receive data on a
The recv() function blocks until data is
available, or until the connection is closed.
It’s passed a size,
It’s passed a buffer, buf, and the
size of the buffer, BUFLEN, and it
reads up to BUFLEN bytes of data.
And what it returns is the number
of bytes of data that were read.
Or, if the connection was closed,
it returns zero. Or, if an error
occurs, it returns -1, and again sets
global variable errno to indicate what happened.
When a recv() call finishes, you have
to check these three possibilities. You have
to check if the return value is
zero, to indicate that the connection is
closed and you've successfully received all the
data in that connection. At which point,
you should also close the connection.
You have to check if the return
value is minus one, in which case
an error has occurred, and that connection
has failed, and you need to somehow
handle that error.
And you need to check if it's some other value,
to indicate that you've received some data,
and then you need to process that data.
What's important is to remember that the
recv() call just gives you that data
in the buffer. If the return value
from receive is 157, this indicates that
the buffer has 157 bytes of data in it.
What the recv() called doesn't ever do,
is add a terminating null to that buffer.
Now, if you're careful that doesn't matter,
because you know how much data is
in the buffer, and you can explicitly
process the data up to that length.
But, a common problem with TCP-based applications,
is that they treat the data as if it was a string.
They pass it to the printf() call
using %s as if it were a
string, or they pass it to function
like strstr() to search for a string
within it, or strcpy(), or something like that.
And the problem is the string functions
assume there’s a terminating null, and the
recv() call doesn't provide one.
If you're going to pass the data
that's returned from a recv() call to
one of the C string functions,
you need to explicitly add that null yourself.
You need to look at the buffer,
add the null at the end,
after the last byte which was successfully
received. If you don't do, this the
string functions will just run off the end of the buffer
and you'll get a buffer overflow attack.
And this is a significant security risk.
It’s one of the biggest security problems
with network code using C. It’s misusing
these buffers, accidentally using one of the
string functions, and it just reads off
the end of buffer, and who knows what it processes.
When you send data using TCP,
the send() call enqueues the data for transmission.
The operating system, the TCP code in
the operating system, splits the data you've
written using the various send() calls into
what’s known as segments, and puts each
of these into a TCP packet.
The TCP packets are sent in IP
packets. And TCP runs a congestion control
algorithm to decide when it can send those packets.
Each TCP segment, each segment is in
a TCP packet. The TCP packets have
a header, which has a sequence number.
When the connection setup handshake happens,
in the SYN and the SYN-ACK packets,
the connection agrees the initial sequence numbers;
agrees the starting value for the sequence numbers.
If you’re the client, for example;
the client picks a sequence number at
random, and sends this in its SYN packet.
And then when it starts sending data,
the next data packet has a sequence
number that is one higher than that
in the SYN packet.
And, as it continues to send data,
the sequence numbers increase by the number
of data bytes sent.
So, for example, if the initial sequence
number was 1001, just picked randomly,
and it sends 30 bytes of data
in the packet, then the next sequence
number will be 1031.
The sequence number spaces are separate for
each in each direction. The sequence numbers
the client uses increase based on the
initial sequence number the client sent the SYN packet.
The sequence numbers the server use,
start based on the initial sequence number
the server sent in the SYN-ACK packet,
and increase based on the amount of
data the server is sending. The two
number spaces are unrelated.
What's important is that calls to send()
don't map directly onto TCP segments.
If the data which is given to
a send() call is too big to
fit into one TCP segment, then the
TCP code will split it across several
segments; it'll split it across several packets.
Similarly, if the data you send,
that data you give the send() call
is quite small, TCP might not send
It might buffer it up, combine it
with data sent as part of a
later send() call. And combine it,
and send it in a single larger
segment, a single larger TCP packet.
This is an idea known as Nagle’s
algorithm. It's there to improve efficiency by
only sending big packets, because there's a
certain amount of overhead for each packet.
Each packet that’s sent by TCP has
a TCP header. It’s got an IP
header. It's got the Ethernet or the
WiFi headers depending on the link layer.
And that adds a certain amount of
overhead. It’s about, I think, 40 bytes
per packet. So if you're only sending
a small amount of data, that's a
lot of overhead, a lot of wasted data.
So TCP, with the Nagle algorithm,
tries to combine these packets into larger
packets when it can. But, of course,
this adds some delay. It’s got to
wait for you to send more data;
wait to see if it can form a bigger packet.
If you really need low latency,
you can disable the Nagle algorithm.
There’s a socket option called TCP_NODELAY,
and we see the code on the
slide to show how to use that.
So you create the socket, you
establish the connection, and then you call
the TCP_NODELAY option and that turns this
off. And this means that every time
you send() on the socket, it immediately
gets sent as quickly as possible.
One implication of this behaviour, though,
where TCP can either split data written
in a single send() across multiple segments,
or where it can combine several send()
calls into a single segment, is that
the data returned by the recv() calls
doesn't always correspond to a single send().
When you call recv(), you might get
just part of a message. And you
need to call recv() again to get the rest of the message.
Or you may get several messages in one recv() call.
When you're using TCP, the recv() calls
return the data reliably, and they return
the data in the order that it was sent.
But what they don't do is frame
the data. What they don't do is
preserve the message boundaries.
For example, if we're using HTTP,
which we see, we see an example
of an HTTP message that might be sent,
an HTTP response that might be sent,
by a web server back to a browser.
If we're using HTTP, what we would
like is that the whole response is
received in one go. So if we're
implementing a web browser we just call
recv() on the TCP connection
and we get all of the headers,
and all of the body, in just
in just one call to recv() and
we can then parse it, and process it, and deal with it.
TCP doesn't guarantee this, though.
It can split the messages arbitrarily,
depending on how much data was in
the packets, what size packets the underlying
link layers can send, and on the
available capacity of the network depending on
the congestion control.
And it can split the packets at arbitrary points.
For example, if we look at the
slide, we see that the headers,
some of them are labeled in red,
some are in blue, some of the body is in blue,
some the rest of the body is
in green. And it could be that
the TCP connection splits the data up,
so that the first recv() call just
gets the part of the headers highlighted
ending halfway through the “ETag:” line.
And then you have to call recv()
again. And then you get the part
of the message highlighted in blue,
which contains the rest of the headers
and the first part of the body.
Then you have to call recv() again,
to get the rest of the message
that's highlighted in green on the slide.
And this makes it much harder to
parse; much harder for the programmer.
Because you have to look at the
data you've got, parse it, check to
see if you've got the whole message,
check if you've received the complete headers,
check to see if you've received the
complete body. And you have to handle
the fact that you might have partial messages.
And it's something which makes it a
little bit hard to debug, because if
you only send small messages,
if you're sending packets which are only
like 1000 bytes, or so, they’re probably
small enough to fit in a single
packet, and they always get delivered in one go.
It’s only when you start sending
larger packets, or sending lots of data
over connection so things get split up
due to congestion control, that you start
to see this behaviour where the messages
get split at arbitrary points.
So as we've seen, the TCP segments
contain sequence numbers, and the sequence numbers
count up with the number of bytes being sent.
Each TCP segment also has an acknowledgement number.
When a TCP segment is sent,
it acknowledges any segments that have previously
if a TCP endpoint has received some
data on a TCP connection,
when it sends its next packet,
the ACK bit will be set in
the TCP header, to indicate that the
acknowledgement number is valid, and the acknowledgement
number will have a value indicating the
next sequence number it is expecting.
That is, the next contiguous byte it's
expecting on the connection.
So, in the example, we have a
slightly unrealistic example in that the connection
is sending one byte at a time,
and the first packet is sent with sequence number five.
And then the next packet is sent
with sequence number six, and then seven,
and eight, and nine, and ten,
and so on. And this is what
might happen with an ssh connection,
where each key you type generates a
TCP segment, with just the one key press in it.
And when those packets are received at
host B, it sends a TCP segment
with the acknowledgement bit set, acknowledging what's
So when it receives the TCP packet
with sequence number five, and one byte
of data in it, it sends an
acknowledgement saying it got it, and it's
expecting the packet with sequence number six next.
When it receives the packet with sequence
number six, and one byte of data
in it, it sends an acknowledgement saying
it's expecting seven. And so on.
TCP only ever acknowledges the next contiguous
sequence number expected.
And if a packet is lost,
subsequent packets generate duplicate acknowledgments.
So in this case, packet five was
sent. It got to the receiver,
and that sent the acknowledgement saying it
expected six. Six was sent, arrived at
the receiver, so the acknowledgement says it
Seven was sent, arrives at the receiver,
sends the acknowledgement saying it expects
eight. Eight was sent, and gets lost.
Nine was sent, and arrives at the receiver.
At this point, the receiver’s received the
packets with sequence numbers five, six,
and seven; eight is missing; and nine
has arrived. So the next contiguous sequence
number it's expecting is still eight.
So it sends an acknowledgement saying “I’m
expecting sequence number eight next”.
The packet sent, the next packet sent,
has sequence number 10. This arrives,
the acknowledgement goes back saying “I still
haven't got eight, I’m still expecting eight”,
and this carries on. TCP keeps sending
duplicate acknowledgments while there’s a gap in
the sequence number space.
In addition, we don't show it here,
but TCP can also send delayed acknowledgments,
where it only acknowledges every second packet.
In this case the acknowledgments might go,
six, eight. The packet with sequence number
five is sent, and it acknowledges six.
Packet with number six is sent,
and arrives, and packet number seven is
sent, and then it sends the acknowledgement
saying it's expecting eight. So it doesn't
have to send every acknowledgement, it can
sent every other acknowledgement to reduce the overheads.
TCP uses the acknowledgments to detect packet
loss; to detect when segments are lost.
There’s two ways in which it does this.
The first is that if it sends
data, but for some reason the acknowledgments stop entirely.
This is a sign that either the receiver has failed,
And, you know, the packets are being
delivered to the receiver, but the application
has crashed, and there's nothing there to
receive the data, to reply.
Or it's an indication that the network
connection has failed, and the packets are
just not reaching the receiver.
So if TCP is sending data,
and it's not getting any acknowledgments back,
after a while it times out and
uses this as an indication that the
connection has failed.
Alternatively, it can be sending data,
and if some data is lost,
but the later segments arrive, then TCP
will start sending the duplicate acknowledgments.
Again, back to the example, we see
that packet eight is lost, packet nine
arrives, and the sequence number, the acknowledgement
number, comes back says “I’m expecting sequence
And packet ten is sent and it
arrives, and it still says “I’m still
expecting packet with sequence number eight”,
and this just carries on.
And, eventually, TCP gets what's known as
a triple duplicate acknowledgement. It’s got the
original acknowledgement saying it's expecting packet eight,
and then three duplicates following that,
so four packets in total, all saying
“I’m still expecting packet eight”.
And what this indicates, is that data
is still arriving, but something's got lost.
It only generates acknowledgements when a new
packet arrives, so if we keep seeing
acknowledgments indicating the same thing, this indicates
that new packets arriving, because that's what
triggers the acknowledgement to be sent,
but there's still a packet missing,
and it's telling us which one it's expecting.
At that point TCP assumes that the
packet has got lost, and retransmits that
segment. It retransmits the packet with sequence
Why does it wait for a triple duplicate acknowledgement?
Why does it not just retransmit it
immediately. when it sees a duplicate?
Well, the example we see here illustrates that.
In this case, a packet with sequence
number five is sent, containing one byte
of data, and it arrives, and the
receiver acknowledges it, saying it's expecting six.
And six is sent, and it arrives,
and the receiver acknowledges it, indicating it’s
And packet seven is sent, and it's
delayed. And packet eight is sent,
and eventually arrives at the receiver.
Now the receiver hasn't received packet seven
yet, so it sends an acknowledgement which
says “I’m still expecting seven”. So that's
a duplicate acknowledgement.
At that point packet seven, which was
delayed, finally does arrive.
Now packet seven has arrived, packet eight
had arrived previously, so what is now
expecting is nine, so it sends an
acknowledgement for nine.
And we see that the acknowledgments go
six, seven, seven, nine, because that packet
seven was delayed a little bit.
And if TCP reacts to a single
duplicate acknowledgement as an indication that the
packet was lost, then you run the
risk that you're resending a packet on
the assumption when it was lost,
when it was just merely delayed a little bit.
And there's a trade off you can make here.
Do you treat, a single duplicate as
an indication of loss? Do you treat
two duplicates as an indication of loss?
Three? Four? Five? At what point do
you say “this as an indication of
loss”, rather than just “this is a
slightly delayed packet, and it might recover
itself in a minute”?
The reason that a triple duplicate is
used, is because someone did some measurements,
and decided that packets being delayed
enough to cause one or two duplicates,
because they arrived just a little bit
out of order, was relatively common.
But packets being delayed enough that they
cause three or more duplicates is rare.
So it's balancing-off speed of loss detection
vs. the likelihood that a merely delayed
packet is treated as if it were
lost, and retransmitted unnecessarily.
And, based on the statistics, the belief
by the designers of TCP was that
waiting for three duplicates was the right threshold.
And you could make a TCP version
that reduced this to two, or even
one duplicate, and it would respond to
loss faster, but would have the risk
that it's more likely to unnecessarily retransmit
something that's just delayed.
Or you could make it four,
five, six, even more duplicate acknowledgments,
which will be less likely to unnecessarily
retransmit data. But it’d be slower,
because it would be slower in responding
to loss, and slower in retransmitting actually lost packets.
The other behaviour of TCP. which is
worth noting, is head-of-line blocking.
Now, in this case we're sending something
more realistic. We're sending full size packets,
with 1500 bytes of data in each packet.
And 1500 is the maximum packet size
that you can send in an Ethernet
packet, or in a WiFi packet,
so this is a typical size that actually gets sent.
In this case, the first packet is
sent with sequence numbers in the range
zero through to 1499.
And this arrives at the receiver,
and the receiver sends an acknowledgement saying
it got it, and the next packet
it’s expecting has sequence number 1500.
So it sends an acknowledgement for 1500.
And if there’s a recv() call outstanding
on that socket, that recv() call will
return at that point, and return 1500
bytes of data. It returns the data
as it was received.
The next packet arrives at the receiver,
containing sequence numbers 1500 through to 2999,
and again the recv() call, if there
is one, will return, and return that
next 1500 bytes.
Similarly, when the packet containing the next
1500 comes in, the receiver will send
the ACK saying “I’m expecting 4500”,
and the recv() call will return.
The packet containing sequence numbers 4500 though
to 5999 is lost.
The packet containing 6000 through to 7499 arrives.
The acknowledgement goes back indicating that it’s
still expecting sequence number 4500, because that
packet got lost. And at that point,
some data has arrived, some new data
has arrived at the receiver.
But there's a gap. The packets,
the packet, containing data with sequence numbers
4500 through to 5999 is still missing.
So if the receiver application has called
recv() on that socket, it won't return.
The data has arrived, it's buffered up
in the TCP layer in the operating
system, but TCP won't give it back
to the application.
And the packets can keep being sent,
and the receiver keeps sending the duplicate
acknowledgments, and eventually it’s sent the triple
duplicate acknowledgement, and the TCP sender notices
and retransmits the packet with sequence numbers
4500 through to 5999.
And eventually those arrive at the receiver.
At that point, the receiver has a
contiguous block of data available, with no
gaps in it, and it returns all
of the data from sequence number 4500
up to sequence number 12,000,
up to the application in one go.
And if the application has given a
big enough buffer, at that point the
recv() call will returned 7500 bytes of
data. It’ll return all of that received
data in one big burst.
And then, as the data, you know,
gets retransmitted, as the data arrives,
it will just keep, you know,
the recv() call will unblock and data
will start flowing.
The point is the TCP receiver waits
for any missing data to be delivered.
If anything's missing, the triple duplicate ACK
happens, it eventually gets retransmitted, and the
receiver won't return anything to the application
until that retransmission has happened.
It’s called head of line blocking.
The data stops being delivered, until it
can be delivered in sequence to the
application. It’s all just buffered up in
the operating system, in the TCP code.
TCP always gives the data to the
application in a contiguous ordered sequence,
in the order it was sent.
And this is another reason why the
recv() calls don't always preserve the message boundaries.
Because it depends how much data was
queued up because of packet losses,
and so on, so that it can
always be delivered in order.
The head of line blocking increases the
total download time. We see on the
left, the case where one packet was
lost, and had to be re-transmitted.
And we see on the right,
the case where all the packets were
received on time. And we see an
increase in the download time because of
the packet loss.
It blocks the receiving, it delays things
a little bit, waiting for the retransmission.
And it increases the overall download time
a little bit.
It disrupts the behaviour of when the
packets are received, during the download quite
significantly. We see 1500, 1500, 1500,
big gap, seven thousand five hundred,
in the case where the packets were
lost. Or, in the case where they
were all received, the data is coming
in quite smoothly. It's regularly spaced.
So it affects the timing, it effects
when the data is delivered to the
application, and it has a smaller effect
on the overall download times.
And if you're building real time applications,
this is a significant problem. We see
the case on the right, if everything
is delivered on time, then the data
is released to the application very quickly
and very predictably.
And you don't need
much buffering delay at the receiver.
Things can be just delivered, things are
just delivered to the application, repeatedly on
a regular schedule.
But the minute something gets lost,
it has to wait for the retransmission.
In this case it waits for one
round trip time, because the ACK has
to get back, and then the data has to be retransmitted.
Plus, it has to wait for four
times the gap between packets, to allow
for the four duplicates, the triple duplicate
ACK and the original ACK, so you
get one round trip time plus four
times the packet spacing.
So if you're using TCP to send,
for example, speech data, where it's sending
packets regularly every 20 milliseconds, you need
to buffer 80 milliseconds plus the round
trip time, to allow for these re-transmissions,
if you're using it for a real time application.
Because, it waits for the retransmissions, and because
of the head of line blocking.
And when you're using applications like Netflix
or the iPlayer, when you press play on the video
there’s a little pause where it says “buffering”.
This is what it's doing. It’s buffering
up enough data that it can wait
for the retransmissions to happen,
buffering up enough data in the TCP
connection that it can keep playing out
the video frames, in order, while still
allowing time for a retransmission to happen.
So it's buffering up the data waiting,
making sure there's enough data buffered up,
because of this head of line blocking
issue in TCP.
So that concludes the discussion of TCP.
It gives you an ordered, reliable, byte stream.
As a service model it's easy to
understand. It’s like reading from a file;
you read from the connection and the
bytes arrive reliably and in the order they were sent.
The timing, though, is unpredictable. How much
you get from the connection each time you read from it,
and whether the data arrives regularly,
or whether it's arrives in big bursts
with large gaps between them, depends on
how much data is lost, and depends
on whether the TCP has to retransmit missing data.
And if you're just using this to
download files that doesn't matter. It means
that the progress bar is perhaps inaccurate,
but otherwise it doesn't make much difference.
But, if you're using it for real
time applications, like video streaming, like telephony,
this head of line blocking can quite
significantly affect the play out.
And a lot of that is the
reason why applications use, why real time
applications use, UDP. And for those that
don't use UDP,
applications like Netflix that use adaptive streaming
over HTTP, which we'll talk about in
lecture seven, that's why there’s this buffering
delay before they start playing.
And, of course, the lack of framing
complicates the application design, you have to
parse the data to make sure you've got all the data;
there's no message boundaries in there,
so you have to parse the data.
It doesn't tell you, the connection doesn't
tell you, when you've received all the data.
So that's it for TCP.
It delivers data reliably. It uses sequence
numbers and acknowledgments to indicate when the
It uses timeouts to indicate that a
connection has failed. And it uses this
idea of triple duplicate ACKs to indicate
that a packet has been lost,
and trigger a retransmission of any lost data.
What I’ll talk about in the next
part is QUIC and how it differs
from the way TCP handles reliability.
The final part of the lecture discusses reliable data transfer using
QUIC. It outlines the QUIC service model, and how it differs from
that of TCP, and shows how QUIC achieves reliable data transfer.
It discusses how QUIC provides multiple streams within a single
connection, and consider how this affects head-of-line blocking
and latency. Approaches to making best use of multiple streams
Slides for part 4
In this final part I’d like to
talk about how reliable data transfer works
with QUIC, and how it's different to
reliable data transfer with TCP.
I’ll talk a little bit about the
QUIC service model, and how it handles
packet numbers and retransmission. I’ll talk about
the multi-streaming features of QUIC. And I’ll
talk about how it avoids head-of-line blocking.
The service model for TCP, as we
saw previously, is that it delivers a
single reliable, ordered, byte stream of data.
Applications write a stream of bytes in,
and that stream of bytes is delivered
to the receiver, eventually.
QUIC, by contrast, delivers several ordered reliable
byte streams within a single connection.
Applications can separate the data they're sending
into different streams, and each stream is
delivered reliably and in order.
QUIC doesn't preserve the ordering between the
streams within a connection, so if you
send in one stream, and then send
in a second stream, then the data
you sent second, in that second stream,
may arrive first, but it preserves the
ordering with a stream.
And you can treat each stream as
if it were running multiple TCP connections
in parallel, so it gives you the
same service model with several streams of
data, or you could perhaps treat each stream as a
sequence of messages to be sent,
with the streams indicating message boundaries.
QUIC delivers data in packets.
Each QUIC packet has a packet sequence
number, a packet number,
and the packet numbers
are split into two packet number spaces.
The packets sent during the initial QUIC
handshake start with packet sequence number zero,
and that packet sequence number increases by
one for each packet sent during the handshake.
Then, when the handshake’s complete, and it
switches to sending data, it resets the
packet sequence number to zero and starts again.
Within each of these packet number spaces,
the handshake space, and the data space,
the packet number sequence starts at zero,
and goes up by one for every packet sent.
That is, the sequence numbers in QUIC,
the packet numbers in QUIC, count the
number of packets of data being sent.
That's different to TCP. In TCP,
the sequence number in the header counts
the offset within the byte stream,
it counts how many bytes of data
have been sent. Whereas in QUIC,
the packet numbers count the number of packets.
Inside a QUIC packet is a sequence
of frames. Some of those frames may
be stream frames, and stream frames carry data.
Each stream frame has a stream ID,
so it knows which of the many sub-streams
it’s carrying data for, and it
also has the amount of data being carried,
and the offset of that data from the start of the stream.
So, essentially the stream contains sequence numbers
which play the same role as TCP
sequence numbers, in that they count bytes
of data being sent in that stream.
And the packets have sequence numbers that
count the number of packets being sent.
And we can see this in the
diagram on the right, where we see
the packet numbers going up, zero,
one, two, three, four. And the stream
numbers, packet zero carries data from the
first stream, bytes zero through 1000.
Packet one carries data from the first
stream, bytes 1001 to 2000. And packet
two carries bytes 2001 to 2500
from the first stream, and zero to
500 from the second stream, and so on.
And we see that we can send
data on multiple streams in a single packet.
QUIC doesn't preserve message boundaries within the
streams. In the same way that,
within a TCP stream, if you write
data to the stream and the amount you write is too big
to fit into a packet, it may
be arbitrarily split between packets.
Or if the data you send in a TCP Stream is too small,
and doesn't fill a whole packet,
it may be delayed waiting for more
data, to be able to fill up
the packet before it’s sent.
The same thing happens with QUIC.
If the amount of data you write to a stream is too big to
fit into a QUIC packet, then it
will be split across multiple packets.
Similarly, if the amount of data you
write to a stream is very small,
QUIC may buffer it up, delay it,
wait for more data, so it can
send it and fill a complete packet.
In addition, QUIC can take data from
more than one stream, and send it
in a single packet, if there’s space to do so.
And if there's more than one stream
with data that's available to send,
then the QUIC sender can make an
arbitrary decision, how it prioritises that data,
and how it delivers frames from each stream.
And usually it will split those,
the data from the streams, so each
packet has data from, half the data from, one stream,
and half from another stream. But it
may alternate them if it wants,
sending one packet with data from stream
1, one from stream 2, one from
stream 1, one from stream 2, and so on.
On the receiving side, the receiver sends,
the QUIC receiver sends acknowledgments for the
packets it receives.
So, unlike TCP which acknowledges the next
expected sequence number, a QUIC receiver just
sends an acknowledgement to say “I got this packet”.
So when packet zero arrives, it sends
an acknowledgement saying “I got packet zero”.
And when packet one arrives, it sends
an acknowledgement saying “I got packet one”, and so on.
The sender needs to remember what data
it puts in each packet, so it
knows when it gets an acknowledgement for packet two that,
in this case, it contained bytes 2001
to 2500 from stream one, and bytes
zero through 500 from stream two.
That information isn't in the acknowledgments.
What's in the acknowledgments it's just the
packet numbers, so the sender needs to
keep track of how it puts the
data from the streams into the packets.
The acknowledgments in QUIC are also a
bit more sophisticated than they are in
TCP, in that it doesn't just have
an acknowledgement number field in the header.
Rather, it sends the acknowledgments as frames
in the packets coming back.
And this gives a lot more flexibility, because
it can have a fairly sophisticated frame
format, and it can change the frame
format to include different, to support different
ways of sending a header, if it needs to.
In the initial version of QUIC,
what's in the frame format, in the
ACK frames coming back from the receiver to the sender,
is a field indicating the largest acknowledgement,
which is essentially the same as the
TCP acknowledgment – it tells you what's
the highest sequence number received.
There's an ACK delay field, that tells
you how long between receiving that packet
the receiver waited before sending the acknowledgement.
So this is the delay in the
receiver. And by measuring the time it
takes for the acknowledgment come back,
and removing this ACK delay field,
you can estimate the network round trip
time excluding the processing delays in the receiver.
There’s a list of ACK ranges.
And the ACK ranges are a way
of the receiver saying “I got a range of packets”.
So you can send an acknowledgement that
says, I got packets from five through seven
in a single go. And you can
split this up, with multiple ACK ranges.
So you could have an acknowledgement that
says “I got packet five; I got packets
seven through nine; and I got packets
11 through 15” and you can send
that all within a single acknowledgement block,
in an ACK frame, within the reverse path stream.
And this gives it more flexibility,
so it doesn't just have to acknowledge
the most recently received packet, which gives
the sender more information to make retransmissions.
This is a bit like the TCP
selective acknowledgement extension.
Like TCP, QUIC will retransmit lost data.
The difference is that TCP retransmits packets,
exactly as they would be originally sent,
so the retransmission looks just the same
as the original packet.
QUIC never retransmits packets.
Each packet in QUIC has a unique packet sequence number,
and each packet is only ever transmitted once.
What QUIC rather does, is it retransmits
the data which was in those packets
in a new packet.
So in this example, we see that
packet, on the slide, we see that
packet number two got lost, and it
contain the data bytes 2001 to 2500
from stream one, and bytes zero through 500 from stream two.
And, when it gets the acknowledgments indicating
that packet was lost, it resends that data.
And in this case it's sending in
packet six, it’s resending the first bytes
of data from stream, it’s sending the
bytes 2001 to 2500 from stream one,
and it will eventually, at some point
later, retransmit the data from stream two.
As we say, each packet has a
unique packet sequence number. Since we're not,
since each packet is acknowledged as it
arrives, and it's not acknowledging the highest,
not acknowledging the next sequence number expected
in the same way TCP does,
you can’t do the triple duplicate ACK
in the same way, because you don't
get duplicate ACKs. Each ACK acknowledges the
next new packet.
Rather QUIC declares a packet to be
lost when it's got ACKS for three
packets with higher packet numbers than the
one which it sent.
At that point, it can retransmit the
data that was in that packet.
And that’s QUIC’s equivalent to the triple
duplicate ACK; it's three following sequence numbers
rather than three duplicate sequence numbers.
And also, just like TCP, if there's
a timeout, and it stops getting ACKs,
then it declares the packets to be lost.
QUIC delivers multiple streams within a single
connection. And within each stream, the data
is delivered reliably, and in the order it was sent.
If a packet’s lost, then that clearly
causes data for the stream, streams,
where the data was included in that packet to be lost.
Whether a packet loss effects one,
or more, streams really depends on how
the sender chooses to put the data
from different streams into the packets.
It’s possible that a QUIC packet can
contain data from several streams. We saw
in the examples, how the packets contain
data from both stream one and stream two simultaneously.
In that case, if a packet is
lost, it will affect both of the
streams, all of the streams if there’s
data from more than two streams in the packet.
Equally, a QUIC sender can choose to
alternate, and send one packet with data
from stream one, and then another packet
with data from stream two, and only
ever put data from a single stream in each packet.
The specification puts no requirements on how
the sender does this, and different senders
can choose to do it differently depending
whether they're trying to make progress on
each stream simultaneously, or whether they want to
they want to alternate, and make sure
that packet loss only ever affects a single stream.
Depending on how they do this,
the streams can suffer from head of
line blocking independently.
If data is lost on a particular
stream, then that stream can't deliver later
data to the application, until that
lost data has been transmitted. But the
other streams, if they've got all the
data, can keep delivering to the application.
So streams suffer from head of line
blocking individually, but there's no head of
line blocking between streams.
This means that the data is delivered
reliably, and in order, on a stream,
but order’s not preserved between streams.
It’s quite possible that one stream can
be blocked, waiting for a retransmission of
some of the data in the packets,
while the other streams are continuing to
deliver data and haven't seen any loss
on that stream.
Each stream is sent and received independently.
And this means if you're careful with how you split data
across streams, and if the implementation is
careful with how it puts data from
streams into different packets, it can limit
the duration of the head of line
blocking, and make the streams independent in
terms of head of line blocking and data delivery.
QUIC delivers, as we've seen, several ordered,
reliable, byte streams of data in a single connection.
How you treat these different bytes streams,
is, I think, still a matter of interpretation.
It's possible to treat a QUIC connection
as though it was several parallel TCP connections.
So, rather than opening multiple TCP connections
to a server, you open one QUIC
connection, and you send and receive several
streams of data within that.
And then you treat each stream of
data as-if it were a TCP stream,
and you parse and process the data
as if it were a TCP stream.
And you possibly send multiple requests,
and get multiple responses, over each stream.
Or, you can treat the streams more as a framing device.
You can say that each stream,
you can choose to interpret each stream,
as sending a single object. And then,
when you send data from the stream,
on that stream, once you finish sending
that object, you close the stream and
move on to use the next one.
And, on the receiving side, you just
read all the data until you see
the end of stream marker, and then
you process it knowing you’ve got a complete object.
And I think that the best practices,
the way of thinking about a QUIC connection,
and the streams within a connection, is still evolving.
And it's not clear which of these
two approaches is the necessarily the right
way to do it. And I think
it probably depends on the application what
makes the most sense.
So, to conclude for this lecture.
We spoke a little bit about best
effort packet delivery on the Internet,
and why the IP layer delivers data
unreliably, and why it's appropriate to have
a best effort network.
Then we spoke a bit about the different transports.
The UDP transport that provides an unreliable,
but timely, service on which you can
build more sophisticated user space application protocols.
We spoke about TCP, that provides a
reliable ordered stream delivery service. And we
spoke about QUIC, that provides a reliable
ordered delivery service with multiple streams of
data. And it’s clear there’s different services,
different transport protocols, for different needs.
What I want to move on to
next time, is starting to talk about
congestion control and how all these different
transport protocols manage the rate at which they send data.
Lecture 5 discussed reliable data transfer over the Internet. It started
with a discussion of best effort packet delivery, and an explanation of
why it makes sense for the Internet to be designed to be an unreliable
network. Then, it moved on to discuss UDP and how to make applications
and new transport protocols that work on an unreliable network. There's
a trade-off between timeliness and reliability that's important here,
and the lecture gave some examples of this to illustrate why many
real-time applications used UDP.
The bulk of the lecture discussed TCP. It spoke about how TCP sends
acknowledgement for packets, how timeouts and triple-duplicate ACKs
indicate loss, and why a triple-duplicate ACK is chosen as the loss
signal. It also discussed head-of-line blocking, and how the in-order,
single stream, reliable service model of TCP leads to head-of-line
blocking and potential latency.
Finally, It discussed the differences between QUIC and TCP. QUIC
acknowledges packets rather than bytes within a stream, uses ACK
frames rather than an ACK header, and delivers multiple streams
of data, allowing it to avoid head-of-line blocking in many cases.
The focus of the discussion will be on how TCP ensures reliability,
to make sure the mechanism is understood, and on the differences
between the TCP and QUIC service models and how QUIC can improve
latency. We'll also discuss how UDP can form a substrate, to easily
allow new transports, to suit different needs, to be built and