<?xmlversion="1.0" encoding="US-ASCII"?> <!-- edited with XMLSPY v5 rel. 3 U (http://www.xmlspy.com) by Daniel M Kohn (private) -->version='1.0' encoding='UTF-8'?> <!DOCTYPE rfcSYSTEM "rfc2629.dtd"[ <!ENTITY nbsp " "> <!ENTITY zwsp "​"> <!ENTITY nbhy "‑"> <!ENTITY wj "⁠"> ]> <rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="exp" docName="draft-irtf-iccrg-rledbat-10"ipr="trust200902"> <?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?> <?rfc toc="yes" ?> <?rfc symrefs="yes" ?> <?rfc sortrefs="yes"?> <?rfc iprnotified="no" ?> <?rfc strict="yes" ?>number="9840" consensus="true" ipr="trust200902" obsoletes="" updates="" submissionType="IRTF" xml:lang="en" tocInclude="true" symRefs="true" sortRefs="true" version="3"> <front> <title abbrev="rLEDBAT">rLEDBAT:receiver-drivenReceiver-Driven Low Extra Delay Background Transport forTCP </title>TCP</title> <seriesInfo name="RFC" value="9840"/> <author fullname="Marcelo Bagnulo" initials="M." surname="Bagnulo"> <organization>Universidad Carlos III de Madrid</organization> <address> <email>marcelo@it.uc3m.es</email> </address> </author> <author fullname="Alberto Garcia-Martinez" initials="A." surname="Garcia-Martinez"> <organization>Universidad Carlos III de Madrid</organization> <address> <email>alberto@it.uc3m.es</email> </address> </author> <!-- [rfced] Alberto, would you prefer that we use accented letters in your name in this and subsequent RFCs? We ask because we see "García-Martínez" in [COMNET1], [COMNET2], and [COMNET3]. We are fine either way, but we ask because some authors prefer that the accents be used. If you prefer that we use the accented letters going forward, we will note your preference for future reference. Original: A. Garcia-Martinez ... Alberto Garcia-Martinez --> <author fullname="Gabriel Montenegro" initials="G." surname="Montenegro"> <address> <email>g.e.montenegro@hotmail.com</email> </address> </author> <author fullname="Praveen Balasubramanian " initials="P." surname="Balasubramanian"> <organization>Confluent</organization> <address> <email>pravb.ietf@gmail.com</email> </address> </author> <date year="2025"/>month="August"/> <workgroup>Internet Congestion Control</workgroup> <!-- [rfced] Please insert any keywords (beyond those that appear in the title) for use on <https://www.rfc-editor.org/search>. --> <!-- [rfced] Please ensure that the guidelines listed in Section 2.1 of RFC 5743 have been adhered to in this document. See <https://www.rfc-editor.org/rfc/rfc5743.html#section-2.1>. --> <abstract> <t> This document specifiesrLEDBAT,receiver-driven Low Extra Delay Background Transport (rLEDBAT) -- a set of mechanisms that enable the execution of a less-than-best-effort congestion control algorithm for TCP at the receiver end. This document is a product of the Internet Congestion Control Research Group (ICCRG) of the Internet Research Task Force (IRTF). </t> </abstract> </front> <middle> <sectiontitle="Introduction">numbered="true" toc="default"> <name>Introduction</name> <t>LEDBAT (Low Extra Delay Background Transport) <xref target="RFC6817"/>format="default"/> is acongestion-controlcongestion control algorithm used for less-than-best-effort (LBE) traffic.</t> <t>When LEDBAT traffic shares a bottleneck with other traffic using standard congestion control algorithms (for example, TCP traffic usingCubic<xrefCUBIC <xref target="RFC9438"/>,format="default"/>, hereafter referred to asstandard-TCP"standard-TCP" for short), it reduces its sending rate earlier and more aggressively than standard-TCP congestion control, allowing other non-background traffic to use more of the available capacity. In the absence of competing traffic, LEDBAT aims to makeanefficient use of the available capacity, while keeping the queuing delay within predefinedbounds.</t>bounds. <!-- [rfced] Section 1: Is there a distinction between "standard-TCP" and "standard TCP" (e.g., "standard TCP sender", "standard-TCP flow") as used in this document, or do they mean the same thing? We ask because we see "hereafter referred (to) as standard-TCP for short" in the second paragraph of Section 1. If "standard-TCP" and "standard TCP" mean the same thing, we suggest removing the hyphen*. * Please note that we also see "standard TCP" but not "standard-TCP" in RFC 6817, and the only published RFC to date that uses "standard-TCP" appears to be RFC 1687 ("A Large Corporate User's View of IPng"), published in August 1994. Original: When LEDBAT traffic shares a bottleneck with other traffic using standard congestion control algorithms (for example, TCP traffic using Cubic[RFC9438], hereafter referred as standard-TCP for short), it reduces its sending rate earlier and more aggressively than standard-TCP congestion control, allowing other non-background traffic to use more of the available capacity. ... rLEDBAT assumes that the sender is a standard TCP sender. ... This guarantees that the rLEDBAT flow will never transmit more aggressively than a standard-TCP flow, as the sender's congestion window limits the sending rate. --> </t> <t>LEDBAT reactsbothto both packet loss andtovariations in delay. With respect to packet loss, LEDBAT reacts with a multiplicative decrease, similar to most TCP congestion controllers. Regarding delay, LEDBAT aims for a targetqueueingqueuing delay. When the measured currentqueueingqueuing delay is below the target, LEDBAT increases the sendingraterate, and when the delay is above the target, it reduces the sending rate. LEDBAT estimates the queuing delay by subtracting the measured current one-way delay from the estimated base one-way delay(i.e.(i.e., the one-way delay in the absence of queues). </t> <t>The LEDBAT specification <xref target="RFC6817"/>format="default"/> defines the LEDBATcongestion-controlcongestion control algorithm, implemented in the sender to control its sending rate. LEDBAT is specified in aprotocolprotocol-agnostic andlayer agnosticlayer-agnostic manner.</t> <t>LEDBAT++ <xref target="I-D.irtf-iccrg-ledbat-plus-plus"/>format="default"/> is also an LBE congestion control algorithmwhichthat is inspired by LEDBAT while addressing several problems identified with the original LEDBAT specification. Inparticularparticular, the differences between LEDBAT and LEDBAT++include: i) LEDBAT++include the following:</t> <ol spacing="normal" type="%i)"> <li>LEDBAT++ uses theround-trip-timeround-trip time (RTT) (as opposed to theone wayone-way delay used in LEDBAT) to estimate the queuingdelay; ii) LEDBAT++delay.</li> <li>LEDBAT++ uses anAdditive Increase/Multiplicative Decreaseadditive increase/multiplicative decrease algorithm to achieve inter-LEDBAT++ fairness and avoid thelate-comerlatecomer advantage observed inLEDBAT; iii) LEDBAT++LEDBAT.</li> <li>LEDBAT++ performs periodic slowdowns to improve the measurement of the basedelay; iv) LEDBAT++delay.</li> <li>LEDBAT++ is defined forTCP.</t>TCP.</li> </ol> <t>In this specification, we describerLEDBAT,receiver-driven Low Extra Delay Background Transport (rLEDBAT) -- a set of mechanisms that enable the execution of an LBE delay-based congestion control algorithm such as LEDBAT or LEDBAT++ at the receiver end of a TCP connection.</t> <t> The consensus of the Internet Congestion Control Research Group (ICCRG) is to publish this document to encourage further experimentation and review of rLEDBAT. This document is not an IETF product and is nota standard.an Internet Standards Track specification. The status of this document isexperimental.Experimental. Insection 4 titled Experiment Considerations,<xref target="sect-5" format="default"/> ("<xref target="sect-5" format="title"/>"), we describe the purpose of the experiment and its current status. </t> </section> <sectiontitle="Motivationsnumbered="true" toc="default"> <name>Conventions and Terminology</name> <t>The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>", "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as described in BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when, they appear in all capitals, as shown here.</t> <t>We use the following abbreviations throughout the text and include them here for the reader's convenience:</t> <dl spacing="normal" newline="false"> <dt>RCV.WND:</dt><dd>The value included in the Receive Window field of the TCP header (which computation is modified by this specification). <!-- [rfced] Appendix A (moved to Section 2, as noted below): a) Please note the following: * Because we found "RFC 2119 key words" (e.g., "MUST", "SHOULD") in this document, per our standard process we added the appropriate boilerplate text and Normative Reference listings. * We moved the contents of Appendix A to a new Section 2, so that readers can read the definitions of the terms before they are used in this document (e.g., "RCV.WND" in Section 4.1). b) We had trouble following the meaning of "(which computation is modified by this specification)". Does "which computation" mean "the computation of which", and does "this specification" refer to this document or the specification of the value? If the suggested text is not correct, please clarify. Original: RCV.WND: the value included in the Receive Window field of the TCP header (which computation is modified by this specification) Suggested: RCV.WND: The value included in the Receive Window field of the TCP header (the computation of which is modified by its specification). --> </dd> <dt>SND.WND:</dt><dd>The TCP sender's window.</dd> <dt>cwnd:</dt><dd>The congestion window as computed by the congestion control algorithm running at the TCP sender.</dd> <dt>RLWND:</dt><dd>The window value calculated by the rLEDBAT algorithm.</dd> <dt>fcwnd:</dt><dd>The value that a standard RFC793bis TCP receiver calculates to set in the receive window for flow control purposes.</dd> <dt>RCV.HGH:</dt><dd>The highest sequence number corresponding to a received byte of data at one point in time.</dd> <dt>TSV.HGH:</dt><dd>The Timestamp Value (TSval) <xref target="RFC7323" format="default"/> corresponding to the segment in which RCV.HGH was carried at that point in time.</dd> <dt>SEG.SEQ:</dt><dd>The sequence number of the last received segment.</dd> <dt>TSV.SEQ:</dt><dd>The TSval value of the last received segment.</dd> </dl> <!-- [rfced] Appendix A and Section 3.1: Regarding "RFC793bis (TCP) receiver": Should RFC 9293 ("Transmission Control Protocol (TCP)"), which obsoletes RFC 793, be cited in the text as suggested below? Original: fcwnd: the value that a standard RFC793bis TCP receiver calculates to set in the receive window for flow control purposes. ... In order to avoid confusion, we will call fcwnd the value that a standard RFC793bis TCP receiver calculates to set in the receive window for flow control purposes. We call RLWND the window value calculated by rLEDBAT algorithm and we call RCV.WND the value actually included in the Receive Window field of the TCP header. For a RFC793bis receiver, RCV.WND == fcwnd. Suggested: fcwnd: The value that a standard TCP receiver compliant with [RFC9293] calculates to set in the receive window for flow control purposes. ... In order to avoid confusion, we will call fcwnd the value that a standard TCP receiver compliant with [RFC9293] calculates to set in the receive window for flow control purposes. We call RLWND the window value calculated by the rLEDBAT algorithm, and we call RCV.WND the value actually included in the Receive Window field of the TCP header. For a receiver compliant with [RFC9293], RCV.WND == fcwnd. --> </section> <section numbered="true" toc="default"> <name>Motivations forrLEDBAT">rLEDBAT</name> <t>rLEDBAT enables new use cases and new deployment models, fostering the use of LBE traffic. The following scenarios are enabled by rLEDBAT:<list> <t>Content</t> <dl spacing="normal" newline="true"> <dt>Content Delivery Networks (CDNs) and more sophisticated file distributionscenarios: Considerscenarios:</dt> <dd>Consider the case where the source of a file to be distributed (e.g., a software developer that wishes to distribute a software update) would prefer to use LBE anditenables LEDBAT/LEDBAT++ in the servers containing the source file. However, because the file is being distributed through a CDN that does not implement LBE congestion control, the result is that the file transfers originated from CDN surrogates will not be using LBE. Interestingly enough, in the case of the software update, the developer may also control the software performing the download in theclient, theclient (the receiver of thefile,file), but because current LEDBAT/LEDBAT++ are sender-based algorithms, controlling the client is not enough to enable LBE congestion control in the communication.rLEDBAT rLEDBAT would enable the use of an LBE traffic class for file distribution in thissetup. </t> <t>Interferencesetup.</dd> <dt>Interference from proxies and othermiddleboxes: Proxiesmiddleboxes:</dt> <dd>Proxies and other middleboxes are commonplace in the Internet. For instance, in the case of mobile networks, proxies are frequently used. In the case of enterprise networks, it is common to deploy corporate proxies for filtering and firewalling. In the case of satellite links, PerformanceEnhancementEnhancing Proxies (PEPs) are deployed to mitigate the effect ofthelongdelaydelays in a TCP connection. These proxies terminate the TCP connection on both ends and prevent the use of LBE congestion control in the segment between the proxy and the sink of the content, the client. By enabling rLEDBAT, clientswould be able tocan then enable LBE traffic between them and theproxy.</t> <t>Receiver-defined preferences. It is frequent that the bottleneck ofproxy.</dd> <dt>Receiver-defined preferences:</dt> <dd>Frequently, thecommunicationaccess link is theaccess link.communication bottleneck. This is particularly true in the case of mobile devices. It is then especially relevant for mobile devices to properly manage the capacity of the access link. With current technologies, it is possible for the mobile device to use different congestion control algorithms expressing different preferences for the traffic. For instance, a device can choose to use standard-TCP for some traffic andtouse LEDBAT/LEDBAT++ for other traffic. However, this would only affect the outgoingtraffictraffic, since both standard-TCP and LEDBAT/LEDBAT++ aresender-driven.driven by the sender. The mobile device has no means to manage the traffic in thedown-link,downlink, whichisis, in most cases, the communication bottleneck for a typicaleye-ball end-user. rLEDBAT"eyeball" end user. rLEDBAT enables the mobile device to selectively use an LBE traffic class for some of the incoming traffic. For instance, by using rLEDBAT, a user can use regular standard-TCP/UDP for a video stream (e.g.,Youtube)YouTube) and use rLEDBAT for other background filedownload.</t> </list></t>downloads.</dd> </dl> </section> <sectiontitle="rLEDBAT mechanisms">numbered="true" toc="default"> <name>rLEDBAT Mechanisms</name> <t>rLEDBAT provides the mechanisms to implement an LBE congestion control algorithm at thereceiver-endreceiver end of a TCP connection. The rLEDBAT receiver controls the sender's rate through the Receive Window announced by the receiver in the TCP header.</t> <t>rLEDBAT assumes that the sender is a standard TCP sender.rLEDBAT rLEDBAT does not require any rLEDBAT-specific modifications to the TCP sender. The envisioned deployment model for rLEDBAT is that the clients implement rLEDBAT and this enables rLEDBAT in communications withexistentexisting standard TCP senders. In particular, the senderMUST<bcp14>MUST</bcp14> implement <xref target="RFC9293"/>format="default"/> and also <bcp14>MUST</bcp14> implement the TCP Timestamps (TS) option as defined in <xref target="RFC7323" format="default"/>. Also, the sender should implement some of the standard congestion control mechanisms, such as CUBIC <xref target="RFC9438" format="default"/> or NewReno <xref target="RFC5681" format="default"/>. <!-- [rfced] Sections 3, 3.2.1, and 3.2.2: a) We changed "Time Stamp Option", "Time Stamp (TS) option", and "TimeStamp option" to "TCP Timestamps option" or "TS option", per RFC 7323 and "TS option generation rules [RFC7323]" used elsewhere in this document. Please let us know any concerns. Original: In particular, the sender MUST implement [RFC9293] and it also MUST implement the Time Stamp Option as defined in<xref target="RFC7323" />.[RFC7323]. ... In order to measure RTT, the rLEDBAT client MUST enable the Time Stamp (TS) option [RFC7323]. ... In the case of TCP, the receiver can use the TimeStamp option to measure the one way delay by subtracting the timestamp contained in the incoming packet from the local time at which the packet has arrived. Currently: In particular, the sender MUST implement [RFC9293] and also MUST implement the TCP Timestamps (TS) option as defined in [RFC7323]. ... In order to measure RTT, the rLEDBAT client MUST enable the TS option [RFC7323]. ... In the case of TCP, the receiver can use the TS option to measure the one-way delay by subtracting the timestamp contained in the incoming packet from the local time at which the packet has arrived. b) We do not see "New Reno", "NewReno", or "Reno" mentioned anywhere in RFC 5681. May we also cite RFC 6582 ("The NewReno Modification to TCP's Fast Recovery Algorithm"), which obsoletes RFC 3782 (which we see mentioned in RFC 5681), for ease of the reader? Original: Also, the sender should implement some of the standard congestion control mechanisms, such as Cubic<xref target="RFC9438" />[RFC9438] or New Reno<xref target="RFC5681" />.[RFC5681]. Suggested: Also, the sender should implement some of the standard congestion control mechanisms, such as CUBIC [RFC9438] or NewReno [RFC5681] [RFC6582]. ... [RFC6582] Henderson, T., Floyd, S., Gurtov, A., and Y. Nishida, "The NewReno Modification to TCP's Fast Recovery Algorithm", RFC 6582, DOI 10.17487/RFC6582, April 2012, <https://www.rfc-editor.org/info/rfc6582>. --> </t> <t>rLEDBAT does not define a new congestion control algorithm. The LBE congestion control algorithm executed in the rLEDBAT receiver is defined in other documents. The rLEDBAT receiverMUST<bcp14>MUST</bcp14> use an LBE congestion control algorithm. Because rLEDBAT assumes a standard TCP sender, the sender will be using a "best effort" congestion control algorithm (such asCubicCUBIC orNew Reno).NewReno). Since rLEDBAT uses the Receive Window to control the sender's rate and the sender calculates the sender's window as the minimum of the Receive window and the congestion window, rLEDBAT will only be effective as long as the congestion control algorithm executed in the receiver yields a smaller window than the one calculated by the sender. This is normally the case when the receiver is using an LBE congestion control algorithm. The rLEDBAT receiverSHOULD<bcp14>SHOULD</bcp14> use the LEDBAT congestion control algorithm <xref target="RFC6817"/>format="default"/> or the LEDBAT++ congestion control algorithm <xref target="I-D.irtf-iccrg-ledbat-plus-plus"/>.format="default"/>. The rLEDBATMAY<bcp14>MAY</bcp14> use other LBE congestion control algorithms defined elsewhere. Irrespective of which congestion control algorithm is executed in the receiver, an rLEDBAT connection will never be more aggressive thanstandard-TCPstandard-TCP, since it is always bounded by the congestion control algorithm executed at thesender.</t>sender. <!-- [rfced] Section 3: a) Will "other documents" be clear to readers? Should one or more specific documents be cited here? Original: The LBE congestion control algorithm executed in the rLEDBAT receiver is defined in other documents. b) Does "The rLEDBAT MAY use other LBE congestion control algorithms defined elsewhere" mean "The rLEDBAT receiver MAY use other LBE congestion control algorithms defined elsewhere" or something else? We ask because we see "the rLEDBAT node", "the rLEDBAT receiver", "the rLEDBAT host", etc. We have the same question re. "the rLEDBAT in host A" (Section 3.2.1.1) and "How the rLEDBAT should resume" (Section 4). Original: The rLEDBAT MAY use other LBE congestion control algorithms defined elsewhere. ... This limitation of the sender's window can come either from the TCP congestion window in host B or from the announced receive window from the rLEDBAT in host A. ... - How the rLEDBAT should resume after a period during which there was no incoming traffic and the information about the rLEDBAT state information is potentially dated. --> </t> <t>rLEDBAT is essentially composed of three types of mechanisms,namely,namely those that provide the means to measure the packet delay (either theround trip timeRTT or theone wayone-way delay, depending on the selected algorithm), mechanisms to detect packetlossloss, and the means to manipulate the Receive Window to control the sender's rate. Theformerfirst two provide input to the LBE congestion controlalgorithmalgorithm, while thelatterthird uses the congestion window computed by the LBE congestion control algorithm to manipulate the Receive window, as depicted inthe figure.</t><xref target="fig1"/>.</t> <figuretitle="Theanchor="fig1"> <name>The rLEDBATarchitecture.">Architecture</name> <artworkalign="center"><![CDATA[align="center" name="" type="" alt=""><![CDATA[ +------------------------------------------+ | TCPreceiverReceiver | | +-----------------+ | | | +------------+ | | | +---------------------| RTT | | | | | | | Estimation | | | | | | +------------+ | | | | | | | | | | +------------+ | | | | +--------------| Loss, RTX | | | | | | | | Detection | | | | | | | +------------+ | | | v v | | | | +----------------+ | | | | | LBE Congestion | | rLEDBAT | | | | Control | | | | | +----------------+ | | | | | | +------------+ | | | | | |RCV-WNDRCV.WND | | | | +---------------->| Control | | | | | +------------+ | | | +-----------------+ | +------------------------------------------+ ]]></artwork> </figure> <t>We next describe each of the rLEDBATcomponents next.</t>components.</t> <sectiontitle="Controllingnumbered="true" toc="default"> <name>Controlling thereceive window">Receive Window</name> <t>rLEDBAT uses the TCP Receive Window (RCV.WND)of TCPto enable the receiver to control the sender's rate. <xref target="RFC9293"/> definesformat="default"/> specifies that the RCV.WND is used to announce the available receive buffer to the sender for flow control purposes. In order to avoid confusion, we will call fcwnd the value that a standard RFC793bis TCP receiver calculates to set in the receive window for flow control purposes. We call RLWND the window value calculated by the rLEDBATalgorithmalgorithm, and we call RCV.WND the value actually included in the Receive Window field of the TCP header. Foraan RFC793bis receiver, RCV.WND == fcwnd.</t> <t>In the case of the rLEDBAT receiver, this receiver <bcp14>MUST NOT</bcp14> set the RCV.WND to a value larger than fcwnd and <bcp14>SHOULD</bcp14> set the RCV.WND to the minimum of RLWND and fcwnd, honoring both. <!-- [rfced] Sections 3.1 and 3.1.1: We had trouble following the meaning of "honoring both", "may fall short to honor", "honoring that", and "sufficient to honor the window output" in these sentences. Please clarify. Original: This may fall short to honor the new calculated value of the RLWND immediately. However, the receiver SHOULD progressively reduce the advertised RCV.WND, always honoring that the reduction is less or equal than the received bytes, until the target window determined by the rLEDBAT algorithm is reached. ... In the case of rLEDBAT receiver, the rLEDBAT receiver MUST NOT set the RCV.WND to a value larger than fcwnd and it SHOULD set the RCV.WND to the minimum of RLWND and fcwnd, honoringboth.</t>both. ... In order to avoid window shrinking, the receiver MUST only reduce RCV.WND by the number of bytes upon of a received data packet. This may fall short to honor the new calculated value of the RLWND immediately. However, the receiver SHOULD progressively reduce the advertised RCV.WND, always honoring that the reduction is less or equal than the received bytes, until the target window determined by the rLEDBAT algorithm is reached. This implies that it may take up to one RTT for the rLEDBAT receiver to drain enough in-flight bytes to completely close its receive window without shrinking it. This is sufficient to honor the window output from the LEDBAT/LEDBAT++ algorithms since they only allow to perform at most one multiplicative decrease per RTT. --> </t> <t>When using rLEDBAT, two congestion controllers are in action in the flow of data from the sender to the receiver,namely,namely the TCP congestion control algorithmof TCP inon the sender side and the LBE congestion control algorithm executed in the receiver and conveyed to the sender through the RCV.WND. In the normal TCP operation, the sender uses the minimum of thecongestion windowcwnd and thereceiver windowRCV.WND to calculate thesender's windowSND.WND. This is also true for rLEDBAT, as the sender is a regular TCP sender. This guarantees that the rLEDBAT flow will never transmit more aggressively than a standard-TCP flow, as the sender's congestion window limits the sending rate. Moreover, becauseaan LBE congestion control algorithm such as LEDBAT/LEDBAT++ is designed to react earlier and more aggressively to congestion than regular TCP congestion control, the RLWND contained in the TCP RCV.WND fieldof TCPwill generally bein generalsmaller than the congestion window calculated by the TCP sender, implying that the rLEDBAT congestion control algorithm will be effectively controlling the sender's window. One exception to this scenario is that at the beginning of the connection, when there is no information to set RLWND, RLWND is set to its maximum value, so that the sending rate of the sender is governed by the flow control algorithm of the receiver and the TCP slow start mechanism of the sender. <!-- [rfced] Section 3.1: We had trouble parsing this sentence. We updated it as follows. If this is incorrect, please clarify the text. Original: One exception to this is at the beginning of the connection, when there is no information to set RLWND, then, RLWND is set to its maximum value, so that the sending rate of the sender is governed by the flow control algorithm of the receiver and the TCP slow start mechanism of thesender.</t>sender. Currently: One exception to this scenario is that at the beginning of the connection, when there is no information to set RLWND, RLWND is set to its maximum value, so that the sending rate of the sender is governed by the flow control algorithm of the receiver and the TCP slow start mechanism of the sender. --> </t> <t>In summary, the sender's windowis:is SND.WND = min(cwnd, RLWND, fcwnd)</t> <sectiontitle="Avoiding window shrinking">numbered="true" toc="default"> <name>Avoiding Window Shrinking</name> <t>The LEDBAT/LEDBAT++ algorithm executed in a rLEDBAT receiver increases or decreases RLWND according to congestion signals (variationsonin the estimatedqueueingqueuing delay and packet loss). If RLWND is decreased and directly announced in RCV.WND, this could lead to an announced window that is smaller than what is currently in use. Thisso called 'shrinkingso-called "shrinking thewindow'window" is discouraged as per <xref target="RFC9293"/>,format="default"/>, as it may cause unnecessary packet loss and performancepenalty.penalties. To be consistent with <xref target="RFC9293"/>,format="default"/>, the rLEDBAT receiverSHOULD NOT<bcp14>SHOULD NOT</bcp14> shrink the receive window. </t> <t>In order to avoid window shrinking, the receiverMUST<bcp14>MUST</bcp14> only reduce RCV.WND by the number of bytes upon of a received data packet. This may fall short to honor the new calculated value of the RLWND immediately. However, the receiverSHOULD<bcp14>SHOULD</bcp14> progressively reduce the advertised RCV.WND, always honoring that the reduction is less than or equalthanto the received bytes, until the target window determined by the rLEDBAT algorithm is reached. This implies that it may take up to one RTT for the rLEDBAT receiver to drain enough in-flight bytes to completely close its receive window without shrinking it. This is sufficient to honor the window output from the LEDBAT/LEDBAT++ algorithms, since they only allow to perform at most one multiplicative decrease per RTT. <!-- [rfced] Section 3.1.1: a) Please clarify "upon of" in this sentence. Are some words missing, or should either "upon" or "of" be removed? Original: In order to avoid window shrinking, the receiver MUST only reduce RCV.WND by the number of bytes upon of a received data packet. b) Does "they only allow to perform" mean "they are only allowed to perform", "they only permit performing", or something else? Original: This is sufficient to honor the window output from the LEDBAT/LEDBAT++ algorithms since they only allow to perform at most one multiplicative decrease perRTT.</t>RTT. --> </t> </section> <sectiontitle="Settingnumbered="true" toc="default"> <name>Setting the Window ScaleOption">Option</name> <t>The Window Scale (WS) option <xref target="RFC7323"/>format="default"/> is a means to increase the maximum window size permitted by the Receive Window. The WS option defines a scale factorwhichthat restricts the granularity of the receive window that can be announced. This means that the rLEDBAT client will have to accumulate the increases resulting from multiple receivedpackets,packets and only convey a change in the window when the accumulated sum of increases is equal to or higher than one increase step as imposed by the scaling factor according to the WS option in place for the TCP connection.</t> <t>Changes in the receive window that are smaller than 1 MSS (Maximum Segment Size) are unlikely to have any immediate impact on the sender'srate, as usualrate. As usual, TCP's segmentation practice results in sending full segments (i.e., segments of size equal to the MSS).Current WS option specification<xref target="RFC7323"/>format="default"/>, which defines the WS option, specifies that allowed values for the WS option are between 0 and 14. Assumingaan MSS of around 1500 bytes, WS option values between 0 and 11 result in the receive window being expressed in units that are about 1 MSS or smaller. So, WS option values between 0 and 11 have no impact in rLEDBAT (unless packets smaller than the MSS are being exchanged).</t> <t>WS option values higher than 11 can affect the dynamics of rLEDBAT, since control may become too coarse (e.g., with a WS option value of 14, a change in one unit of the receive window implies a change of 10 MSS in the effectivewindow).</t>window). <!-- [rfced] Section 3.1.2: We changed "with WS of 14" to "with a WS option value of 14" here, to indicate the option value as opposed to the concept of window scale. If this is incorrect, please clarify. Original: WS option values higher than 11 can affect the dynamics of rLEDBAT, since control may become too coarse (e.g., with WS of 14, a change in one unit of the receive window implies a change of 10 MSS in the effective window). Currently: WS option values higher than 11 can affect the dynamics of rLEDBAT, since control may become too coarse (e.g., with a WS option value of 14, a change in one unit of the receive window implies a change of 10 MSS in the effective window). --> </t> <t>For the above reasons, the rLEDBAT clientSHOULD<bcp14>SHOULD</bcp14> set WS option values lower than 12. Additional experimentation is required to explore the impact of larger WS values on rLEDBAT dynamics.</t> <t>Note that the recommendation for rLEDBAT to set the WS optionvaluevalues to lower values does notprecludes thepreclude communication with servers that set the WS option values to larger values, sincetheWS optionvalue isvalues are set independently for each direction of the TCP connection.</t> </section> </section> <sectiontitle="Measuring delays">numbered="true" toc="default"> <name>Measuring Delays</name> <t>Both LEDBAT and LEDBAT++ measure base and current delays to estimate thequeueingqueuing delay. LEDBAT uses theone way delayone-way delay, while LEDBAT++ uses theround trip time.RTT. In the nextsectionssections, we describe how rLEDBAT mechanisms enable the receiver to measure theone wayone-way delay or theround trip time, whateverRTT -- whichever isneededneeded, depending on the congestion control algorithm used.</t> <sectiontitle="Measuringnumbered="true" toc="default"> <name>Measuring RTT toestimateEstimate thequeueing delay">Queuing Delay</name> <t>LEDBAT++ uses theround trip time (RTT)RTT to estimate thequeueingqueuing delay. In order to estimate thequeueingqueuing delay using RTT, the rLEDBAT receiver estimates the base RTT (i.e., the constant components of RTT) and also measures the current RTT. By subtracting these two values, we obtain the queuing delay to be used by the rLEDBAT controller.</t> <t>LEDBAT++ discovers the base RTT (RTTb) by taking the minimum value of the measured RTTs over a period of time. The current RTT (RTTc) is estimated using a number of recent samples and applying a filter, such as the minimum (or the mean) of the last k samples. Using RTT to estimate thequeueingqueuing delay has a number of shortcomings anddifficulties that we discuss next.</t>difficulties, as discussed below.</t> <t>The queuing delay measured using RTTincludesalso includes thequeueingqueuing delay experienced by the return packets in the direction from the rLEDBAT receiver to the sender. This is a fundamental limitation of this approach. The impact of this error is that the rLEDBAT controller will also react to congestion in the reverse path direction, resulting in an even more conservative mechanism. <!-- [rfced] Section 3.2.1: Please confirm that "error" is the correct word here. The approach discussed in this section does not seem to otherwise be considered an error - only an approach with a limitation (per the previous sentence). Please confirm that calling this approach an error will be clear to readers. Original (the previous sentence is included for context): This is a fundamental limitation of this approach. The impact of this error is that the rLEDBAT controller will also react to congestion in the reverse path direction which results in an even more conservativemechanism.</t>mechanism. Perhaps ("this limitation"): This is a fundamental limitation of this approach. The impact of this limitation is that the rLEDBAT controller will also react to congestion in the reverse path direction, resulting in an even more conservative mechanism. Or possibly ("this issue"): This is a fundamental limitation of this approach. The impact of this issue is that the rLEDBAT controller will also react to congestion in the reverse path direction, resulting in an even more conservative mechanism. --> </t> <t>In order to measure RTT, the rLEDBAT clientMUST<bcp14>MUST</bcp14> enable theTime Stamp (TS)TS option <xref target="RFC7323"/>.format="default"/>. By matching theTSValTSval value carried in outgoing packets with theTSecrTimestamp Echo Reply (TSecr) value <xref target="RFC7323" format="default"/> observed in incoming packets, it is possible to measure RTT. This allows the rLEDBAT receiver to measure RTT even if it is acting as a pure receiver. In a pure receiver, there is no data flowing from the rLEDBAT receiver to the sender, making it impossible to match data packets with Acknowledgment packets to measure RTT, as it is usually done in TCP for other purposes. <!-- [rfced] Section 3.2.1: Does "as it is usually done in TCP" indicate a comparison or a contrast? If the suggested text is not correct, please clarify. Original: In a pure receiver there is no data flowing from the rLEDBAT receiver to the sender, making impossible to match data packets with acknowledgements packets to measure RTT, as it is usually done in TCP for otherpurposes.</t>purposes. Suggested (guessing a contrast): In a pure receiver, there is no data flowing from the rLEDBAT receiver to the sender, making it impossible to match data packets with Acknowledgment packets to measure RTT, in contrast to what is usually done in TCP for other purposes. --> <!-- [rfced] Sections 3.2.1 and subsequent: Because "TSval" stands for "Timestamp Value" per RFC 7323, may we change the instances of "TSval value" to "TSval", to avoid the appearance of "Timestamp Value value"? --> </t> <t>Depending on the frequency of the local clock used to generate the values included in the TS option, several packets may carry the sameTSValTSval value. If that happens, the rLEDBAT receiver will be unable to match the different outgoing packets carrying the sameTSValTSval value with the different incoming packetscarryingalso carrying the same TSecr value. However, it is not necessary for rLEDBAT to use all packets to estimateRTTRTT, and sampling a subset of in-flight packets per RTT is enough to properly assess thequeueingqueuing delay. RTTMUST<bcp14>MUST</bcp14> then be calculated as the time since the first packet with a givenTSValTSval was sent and the first packet that was received with the same value contained in the TSecr. Other packets with repeated TS valuesSHOULD NOT<bcp14>SHOULD NOT</bcp14> be used for RTTcalculation.calculations. </t> <t>Several issues must be addressed in order to avoid an artificial increaseofin the observed RTT. Different issuesemergeemerge, depending on whether therLEDBAT capablerLEDBAT-capable host is sending data packets or pure ACKs to measure RTT. We next considerthethese issues separately.</t> <sectiontitle="Measuringnumbered="true" toc="default"> <name>Measuring RTTsending pure ACKs">When Sending Pure ACKs</name> <t>In this scenario, the rLEDBAT node (node A) sends a pure ACK to the other endpoint of the TCP connection (node B), including the TS option. Upon the reception of the TSOption,option, host B will copy the value of theTSValTSval into the TSecr field of the TS option and include that optionintoin the next data packet towards host A. However, there are two reasons why B may not send a packet immediately back to A, artificially increasing the measured RTT. The first reason is when A has no data to send. The second is when A has no available window to put more packetsin-flight.in flight. Wedescribenext describe how each of these cases is addressed.</t> <t>The case wherethehost B has no data to send when it receives the pureAcknowledgementAcknowledgment is expected to be rare in the rLEDBAT use cases.rLEDBAT rLEDBAT will be used mostly for background filetransferstransfers, so the expected common case is that the sender will have data to send throughout the lifetime of the communication. However, if, for example, the file is structured in blocks of data, it may be the case that the senderseldomlywill seldom have to wait until the next block is available to proceed with the data transfer. To address this situation, the filter used by the congestion control algorithm executed in the receiver <bcp14>SHOULD</bcp14> discard outliers (e.g., a MIN filter <xref target="RFC6817"/> would achieve this) when measuring RTT using pure ACK packets. <!-- [rfced] Sections 3.2.1.1 and 3.2.1.2: For ease of the reader, we changed "min filter" to "MIN filter" and cited RFC 6817 here (where "MIN filter" is first used). Please let us know any concerns. Original: To address this situation, the filter used by the congestion control algorithm executed in the receiver SHOULD discard outliers (e.g. a min filter would achieve this) when measuring RTT using pure ACKpackets.</t>packets. ... Also, applying a filter that discards outliers would also address this issue (e.g. a min filter). Currently: To address this situation, the filter used by the congestion control algorithm executed in the receiver SHOULD discard outliers (e.g., a MIN filter [RFC6817] would achieve this) when measuring RTT using pure ACK packets. ... Applying a filter (e.g., a MIN filter) that discards outliers would also address this issue. --> </t> <t>This limitation of the sender's window can comeeitherfrom either the TCP congestion window in host B orfromthe announced receive window from the rLEDBAT in host A. Normally, the receive window will be the one to limit the sender's transmission rate, since the LBE congestion control algorithm used by the rLEDBAT node is designed to be more restrictive on the sender's rate than standard-TCP. If the limiting factor is the congestion window in the sender, it is less relevant if rLEDBAT further reduces the receive window due to a bloated RTT measurement, since the rLEDBAT node is not actively controlling the sender's rate. Nevertheless, the proposed approach to discard larger samples would also address this issue.</t> <t>To address the case in which the limiting factor is the receive window announced by rLEDBAT, the congestion control algorithm at the receiverSHOULD<bcp14>SHOULD</bcp14> discard RTT measurements during the window reduction phase that are triggered by pure ACK packets. The rLEDBAT receiver is aware of whether a givenTSValTSval value was sent in a pure ACK packet where the window was reduced, and if so, it can discard the corresponding RTT measurement. </t> </section> <sectiontitle="Measuringnumbered="true" toc="default"> <name>Measuring RTTwhen sending data packets">When Sending Data Packets</name> <t>In the case that the rLEDBAT node is sending data packets and matching them with pure ACKs to measure RTT, a factor that can artificially increase the RTT measured is the presence of delayedAcknowledgements.Acknowledgments. According to the TS option generation rules <xref target="RFC7323"/>,format="default"/>, the value included in the TSecr for a delayed ACK is the one in theTSValTSval field of the earliest unacknowledged segment. This may artificially increase the measured RTT. </t> <t>If both endpoints of the connection are sending data packets, Acknowledgments are piggybackedintoonto the data packets and they are not delayed. Delayed ACKs only increase RTT measurements in the case that the sender has no data to send. Since the expected use case for rLEDBAT is that the sender will be sending background traffic to the rLEDBAT receiver, the cases where delayed ACKs increase the measured RTT are expected to be rare.</t> <t>Nevertheless, measurements based on data packets from the rLEDBAT node matching pure ACKs from the other end will result in an increased RTT sample. The additional increase in the measured RTT will be up to 500 ms.The reason for thisThis isthatbecause delayed ACKs are generated every second data packet received and not delayed more than 500 ms according to <xref target="RFC9293"/>.format="default"/>. The rLEDBAT receiverMAY<bcp14>MAY</bcp14> discard RTT measurements done using data packets from therLEBDATrLEDBAT receiver and matching pure ACKs, especially if it has recent measurements done using other packet combinations.Also, applyingApplying a filter (e.g., a MIN filter) that discards outliers would also address thisissue (e.g. a min filter).</t>issue.</t> </section> </section> <sectiontitle="Measuring one way delaynumbered="true" toc="default"> <name>Measuring One-Way Delay toestimateEstimate thequeueing delay">Queuing Delay</name> <t>The LEDBAT algorithm uses the one-way delay of packets as input. A TCP receiver can measure the delay of incoming packets directly (as opposed to the sender-based LEDBAT, where the receiver measures the one-way delay and needs to convey it to the sender).</t> <t>In the case of TCP, the receiver can use theTimeStampTS option to measure theone wayone-way delay by subtracting the timestamp contained in the incoming packet from the local time at which the packet has arrived. As noted in <xref target="RFC6817"/>format="default"/>, the clock offset between the sender's clockof the senderand the receiver's clockin the receiverdoes not affect the LEDBAT operation, since LEDBAT uses the difference between the baseone wayone-way delay and the currentone wayone-way delay to estimate the queuing delay, effectivelycanceling"canceling out" the clock offset error in thequeueingqueuing delay estimation. Thereare howeverare, however, two other issues that the rLEDBAT receiver needs to take into account in order to properly estimate theone wayone-way delay,namely,namely the units in which the received timestamps are expressed and the clock skew. These issues are addressed below. <!-- [rfced] Section 3.2.2: Weaddress them next.</t>changed 'effectively canceling the clock offset error' to 'effectively "canceling out" the clock offset error' per Appendix A.1 of RFC 6817 (which says 'the offsets cancel each other out in the queuing delay estimate'). Please let us know any objections. Original: As noted in [RFC6817] the clock offset between the clock of the sender and the clock in the receiver does not affect the LEDBAT operation, since LEDBAT uses the difference between the base one way delay and the current one way delay to estimate the queuing delay, effectively canceling the clock offset error in the queueing delay estimation. Currently: As noted in [RFC6817], the clock offset between the sender's clock and the receiver's clock does not affect the LEDBAT operation, since LEDBAT uses the difference between the base one-way delay and the current one-way delay to estimate the queuing delay, effectively "canceling out" the clock offset error in the queuing delay estimation. --> </t> <t>In order to measure theone wayone-way delay using TCP timestamps, the rLEDBATreceiver, first,receiver first needs to discover the units of values in the TS optionand, second,and then needs to account for the skew between the two endpoint clocks. Note that a mismatch of 100 ppm (parts per million) in the estimation of the sender's clock rate accounts for 6 ms of variation per minute in the measured delay. This is just one order of magnitude below the target delay set by rLEDBAT (or potentially more if the target is set to lower values, which is possible). Typical skew for untrained clocks is reported to be around 100-200 ppm <xref target="RFC6817"/>.</t>format="default"/>.</t> <t>In order to learn both the TS units and the clock skew, the rLEDBAT receiver measures how much local time has elapsed between two packets with different TS values issued by the sender. By comparing the local time difference and the TS value difference, the receiver can assess the TS units and relative clock skews. In order for this to be accurate, the packets carrying the different TS values should experience equal (or at leastsimilar delay)similar) delay when traveling from the sender to the receiver, as any difference in the experienced delays would introduce an error in the unit/skew estimation. One possible approach is to select packets that experiencedthe minimumminimal delay (i.e., queuing delay(i.e.close tozero queueing delay)zero) to make the estimations.</t> <t>An additional difficulty regarding the estimation of the TS units and clock skew in the context of (r)LEDBAT is that the LEDBAT congestion controller actions directly affect the(queueing)(queuing) delay experienced by packets. In particular, if there is an error in the estimation of the TS units/skew, the LEDBAT controller will attempt to compensate for it by reducing/increasing the load. The result is that the LEDBAT operation interferes with the TS units/clock skew measurements. Because of this, measurements are more accurate when there is no traffic in the connection (in addition to the packets used for the measurements). The problem is that the receiver is unaware if the sender is injecting traffic at any point in time, and so, it is unable to use these quiet intervals to perform measurements. The receiver can, however, force periodic slowdowns, reducing the announced receive window to a few packets and perform the measurements then. <!-- [rfced] Section 3.2.2: We had trouble parsing these sentences. If the suggested text is not correct, please clarify the meaning of "the receiver is unaware if the sender is injecting traffic" and "reducing the announced receive window to a few packets and perform". Original: The problem is that the receiver is unaware if the sender is injecting traffic at any point in time, and so, it is unable to use these quiet intervals to perform measurements. The receiver can however, force periodic slowdowns, reducing the announced receive window to a few packets and perform the measurementsthen.</t>then. Suggested: The problem is that the receiver is unaware of whether the sender is injecting traffic at any point in time; it is therefore unable to use these quiet intervals to perform measurements. The receiver can, however, force periodic slowdowns, reducing the announced receive window to a few packets and performing the measurements at that time. --> </t> <t>It is possible for the rLEDBAT receiver to perform multiple measurements to assess both the TS units and the relative clock skew during the lifetime of the connection, in order to obtain more accurate results. Clock skew measurements are more accurate if the time period used to discover the skew is larger, as the impact of the skew becomes more apparent. It is a reasonable approach for the rLEDBAT receiver to perform an early discovery of the TS units (and the clock skew) using the first few packets of the TCP connection and then improve the accuracy of the TS units/clock skew estimation using periodic measurements later in the lifetime of the connection. </t> </section> </section> <sectiontitle="Detecting packet lossesnumbered="true" toc="default"> <name>Detecting Packet Losses andretransmissions">Retransmissions</name> <t>The rLEDBAT receiver is capable of detecting retransmitted packetsin the following way.as follows. We call RCV.HGH the highest sequence number corresponding to a received byte of data (not assuming that all bytes with smaller sequence numbers have been received already, there may beholes)holes), and we call TSV.HGH theTSValTSval value corresponding to the segment in which that byte was carried. SEG.SEQ stands for the sequence number of a newly receivedsegmentsegment, and we call TSV.SEQ theTSValTSval value of the newly received segment.</t> <t>If SEG.SEQ < RCV.HGH and TSV.SEQ> TSV.HGH> TSV.HGH, then the newly received segment is a retransmission. This is so because the newly received segment was generated later than anotheralready receivedalready-received segmentwhichthat contained data with a larger sequence number. This means that this segment was lost and was retransmitted.</t> <t>The proposed mechanism to detect retransmissions at the receiver fails when there are window tail drops. If all packets in the tail of the window are lost, the receiver will not be able to detect a mismatch between the sequence numbers of the packets and the order of the timestamps. In this case, rLEDBAT will not react to losses but the TCP congestion controller at the sender will, most likely reducing its window to 1 MSS and take over the control of the sending rate, until slow start ramps up and catches the current value of the rLEDBAT window. <!-- [rfced] Section 3.3: This sentence does not parse. If the suggested text is not correct, please clarify "reducing its window to 1MSS and take over the control". Original (the previous sentence is included for context): If all packets in the tail of the window are lost, the receiver will not be able to detect a mismatch between the sequence numbers of the packets and the order of the timestamps. In this case, rLEDBAT will not react to losses but the TCP congestion controller at the sender will, most likely reducing its window to 1MSS and take over the control of the sending rate, until slow start ramps up and catches the current value of the rLEDBATwindow.</t>window. Suggested (the missing space in "1MSS" has been added): In this case, rLEDBAT will not react to losses; however, the TCP congestion controller at the sender will, most likely reducing its window to 1 MSS and taking over the control of the sending rate until slow start ramps up and catches the current value of the rLEDBAT window. --> </t> </section> </section> <sectiontitle="Experiment Considerations">numbered="true" anchor="sect-5" toc="default"> <name>Experiment Considerations</name> <t>The status of this document is Experimental. The general purpose of the proposed experiment is to gain more experience running rLEDBAT over different network paths to see if the proposed rLEDBAT parameters perform well in different situations. Specifically, we would like to learn about the following aspects of the rLEDBAT mechanism: </t><t><list> <t>- Interaction<ul spacing="normal"> <li> <t>Interaction between thesendersender's andthe receiver Congestionreceiver's congestion control algorithms.rLEDBAT rLEDBAT posits that because the rLEDBAT receiver is using a less-than-best-effort congestion control algorithm, thereceiverreceiver's congestion control algorithm will expose a smaller congestion window (conveyedthoughthrough the Receive Window) than the one resulting from the congestion control algorithm executed at the sender. One of the purposes of the experiment is to learn how these two algorithms interact and if the assumption that the receiver side is always controlling the sender's rate (and making rLEDBAT effective) holds. The experiment should include the different congestion control algorithms that are currently widely used in the Internet, includingCubic, BBRCUBIC, Bottleneck Bandwidth and Round-trip propagation time (BBR), and LEDBAT(++). <!-- [rfced] Section 4: We (1) changed "the sender and the receiver Congestion control algorithms" to "the sender's and receiver's congestion control algorithms" per the next sentence andLEDBAT(++).</t> <t>-(2) clarified that "these two" means "these two algorithms". Please let us know if anything is incorrect. Original (the next sentence is included for context): - Interaction between the sender and the receiver Congestion control algorithms. rLEDBAT posits that because the rLEDBAT receiver is using a less-than-best-effort congestion control algorithm, the receiver congestion control algorithm will expose a smaller congestion window (conveyed though the Receive Window) than the one resulting from the congestion control algorithm executed at the sender. One of the purposes of the experiment is learn how these two interact and if the assumption that the receiver side is always controlling the sender's rate (and making rLEDBAT effective) holds. Currently ("conveyed though the" has also been corrected): * Interaction between the sender's and receiver's congestion control algorithms. rLEDBAT posits that because the rLEDBAT receiver is using a less-than-best-effort congestion control algorithm, the receiver's congestion control algorithm will expose a smaller congestion window (conveyed through the Receive Window) than the one resulting from the congestion control algorithm executed at the sender. One of the purposes of the experiment is to learn how these two algorithms interact and if the assumption that the receiver side is always controlling the sender's rate (and making rLEDBAT effective) holds. --> </t> </li> <li> <t>Interaction between rLEDBAT and Active Queue Management techniques such asCodel, PIEControlled Delay (CoDel); Proportional Integral controller Enhanced (PIE); andL4S.</t> <t>- HowLow Latency, Low Loss, and Scalable Throughput (L4S). </t> </li> <li> <t>How the rLEDBAT should resume after a period during which there was no incoming traffic and the information about the rLEDBAT state information is potentially dated.</t></list></t></li> </ul> <sectiontitle="Statusnumbered="true" toc="default"> <name>Status of theexperimentExperiment at thetimeTime ofthis writing."> <t>Currently there areThis Writing</name> <t>Currently, the following implementations of rLEDBATthatcan be used forexperimentation: <list> <t>- Windowsexperimentation:</t> <ul spacing="normal"> <li> <t>Windows 11.rLEDBAT rLEDBAT is available in Microsoft's Windows 11 22H2 since October 2023 <xref target="Windows11"/>.</t> <t>- Windowsformat="default"/>.</t> </li> <li> <t>Windows Server 2022.rLEDBAT rLEDBAT is available in Microsoft's Windows Server 2022 since September 2022 <xref target="WindowsServer"/>.</t> <t>- Apple. rLEDBATformat="default"/>.</t> </li> <li> <t>Apple. rLEDBAT is available inMacOSmacOS and iOS since 2021 <xref target="Apple"/>.</t> <t>-format="default"/>.</t> </li> <li> <t>Linux implementation, open source, available since 2022 at <eref target="https://github.com/net-research/rledbat_module" brackets="angle"/>.</t> </li> <li> <t>ns3 implementation, open source, available since 2020 at <eref target="https://github.com/manas11/implementation-of-rLEDBAT-in-ns-3" brackets="angle"/>. <!-- [rfced] Section 4.1: a) Because the latest version of [Windows11] is dated October 2024 and "2023" is not mentioned on the page, we cannot verify "since October 2023". A Google search for "Windows 11 22H2 ledbat 2023" does not provide any information. Will "October 2023" be clear to readers, or should this item be rephrased? If you would like to rephrase, please provide clarifying text. Original: - Windows 11. rLEDBAT is available in Microsoft's Windows 11 22H2 since October 2023 [Windows11]. b) Would you like us to cite these GitHub pages and list them in the Informative References section, as suggested below? Original: - Linux implementation, open source, available since 2022 athttps://github.com/net-research/rledbat_module.</t> <t>-https://github.com/net-research/rledbat_module. - ns3 implementation, open source, available since 2020 athttps://github.com/manas11/implementation-of-rLEDBAT-in-ns-3.</t> </list></t>https://github.com/manas11/implementation-of-rLEDBAT-in-ns-3. Suggested: * Linux implementation, open source, available since 2022 [rledbat_module]. * ns3 implementation, open source, available since 2020 [rLEDBAT-in-ns3]. ... [rledbat_module] "rledbat_module", commit d82ff20, September 2022, <https://github.com/net-research/rledbat_module>. [rLEDBAT-in-ns3] "Implementation-of-rLEDBAT-in-ns-3", commit 2ab34ad, June 2020, <https://github.com/manas11/ implementation-of-rLEDBAT-in-ns-3>. --> </t> </li> </ul> <t>In addition, rLEDBAT has been deployed by Microsoftinat wide scale in the following services:<list> <t>- BITS</t> <ul spacing="normal"> <li> <t>BITS (Background Intelligent Transfer Service)</t><t>- DO</li> <li> <t>DO (Delivery Optimization) service</t><t>- Windows</li> <li> <t>Windows update # using DO</t><t>- Windows</li> <li> <t>Windows Store # using DO</t><t>- OneDrive</t> <t>- Windows</li> <li> <t>OneDrive</t> </li> <li> <t>Windows Error Reporting # wermgr.exe; werfault.exe</t><t>- System</li> <li> <t>System Center Configuration Manager (SCCM)</t><t>- Windows</li> <li> <t>Windows Media Player</t><t>- Microsoft</li> <li> <t>Microsoft Office</t><t>- Xbox</li> <li> <t>Xbox (download games) # using DO</t></list> </t></li> <!-- [rfced] Section 4.1: Do the "#" symbols mean "number" in these items or something else? Will the text be clear "as is" to readers? If not, please clarify. Original: - Windows update # using DO - Windows Store # using DO ... - Windows Error Reporting # wermgr.exe; werfault.exe ... - Xbox (download games) # using DO --> </ul> <t> Some initial experiments involving rLEDBAT have been reported in <xref target="COMNET3"/>.format="default"/>. Experiments involving the interactionofbetween LEDBAT++ and BBR are presented in <xref target="COMNET2"/>.format="default"/>. An experimental evaluation of the LEDBAT++ algorithm is presented in <xref target="COMNET1"/>.format="default"/>. As LEDBAT++ is one of the less-than-best-effort congestion control algorithms that rLEDBAT relies on, the results regarding how LEDBAT++interactioninteracts with other congestion control algorithms are relevant for the understanding of rLEDBAT as well.</t> </section> </section> <sectiontitle="Security Considerations">numbered="true" toc="default"> <name>Security Considerations</name> <t>Overall, we believe that rLEDBAT does not introduce any new vulnerabilities to existing TCP endpoints, as it relies on existing TCP knobs, notably the Receive Window and timestamps. </t> <t>Specifically, rLEDBAT uses RCV.WND to modulate the rate of the sender. An attacker wishing to starve a flow can simply reduce the RCV.WND, irrespective of whether rLEDBAT is being used or not.</t> <t> We can further ask ourselves whether the attacker can use the rLEDBAT mechanisms in place to force the rLEDBAT receiver to reduce theRCV WND.RCV.WND. There are two ways an attacker can dothat. Onethis:</t> <ul spacing="normal"> <li>One would be to introduce an artificial delay to the packetseitherby either actually delaying the packets or modifying theTimestamps.timestamps. This would cause the rLEDBAT receiver to believe that a queue is building up and reduce the RCV.WND. Note thatan attackerto dothatso, an attacker must be on path, so if that is the case, it is probably more direct to simply reduce theRCV.WND.</t> <t> TheRCV.WND.</li> <li>The other option would be for the attacker to make the rLEDBAT receiver believe that a loss has occurred. To dothat,this, it basically needs to retransmit an old packet (to be precise, it needs to transmit a packet with therightcorrect sequence number and therightcorrect port and IP numbers). This means that the attacker can achieve a reduction of incoming traffic to the rLEDBAT receiver not only by modifying the RCV.WND field of the packets originated from the rLEDBAThost,host but also by injecting packets with the proper sequence number in the other direction. This may slightly expand the attacksurface.</t>surface.</li> </ul> </section> <sectiontitle="IANA Considerations"> <t>No actions are required from IANA.</t> </section> <section title="Acknowledgements">numbered="true" toc="default"> <name>IANA Considerations</name> <t>Thiswork was supported by the EU through the StandICT projects RXQ, CCI and CEL6, the NGI Pointer RIM project and the H2020 5G-RANGE project and by the Spanish Ministry of Economy and Competitiveness through the 5G-City project (TEC2016-76795-C6-3-R).</t> <t>We would like to thank ICCRG chairs Reese Enghardt and Vidhi Goel for their support on this work. We would also like to thank Daniel Havey for his help. We would like to thank Colin Perkins, Mirja Kuehlewind, and Vidhi Goel for their reviews and comments on earlier versions of this document.</t>document has no IANA actions.</t> </section> </middle> <back> <displayreference target="I-D.irtf-iccrg-ledbat-plus-plus" to="LEDBAT++"/> <references> <name>References</name> <referencestitle="Informative References"> <?rfc include='reference.RFC.9293'?> <?rfc include='reference.I-D.irtf-iccrg-ledbat-plus-plus" ?> <?rfc include="reference.RFC.6817" ?> <?rfc include="reference.RFC.7323" ?> <?rfc include="reference.RFC.9438"?> <?rfc include="reference.RFC.5681"?>anchor="sec-normative-references"> <name>Normative References</name> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/> </references> <references anchor="sec-informative-references"> <name>Informative References</name> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9293.xml"/> <!-- draft-irtf-iccrg-ledbat-plus-plus (I-D Exists) --> <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.irtf-iccrg-ledbat-plus-plus.xml"/> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6817.xml"/> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7323.xml"/> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9438.xml"/> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.5681.xml"/> <reference anchor="Windows11">target="https://learn.microsoft.com/en-us/windows/deployment/do/whats-new-do"> <front> <title>What's new in Delivery Optimization</title><author initials="C.F." surname="Forsmann" fullname="Carmen"> <organization /><author> <organization>Microsoft</organization> </author> <dateyear="2023" />month="October" year="2024"/> </front><seriesInfo name="Microsoft Documentation" value="https://learn.microsoft.com/en-us/windows/deployment/do/whats-new-do" /> <refcontent></refcontent><refcontent>Microsoft Windows Documentation</refcontent> </reference> <reference anchor="WindowsServer">target="https://techcommunity.microsoft.com/t5/networking-blog/ledbat-background-data-transfer-for-windows/ba-p/3639278"> <front> <title>LEDBAT Background Data Transfer for Windows</title> <authorinitials="D.H."initials="D" surname="Havey" fullname="Daniel"><organization /><organization/> </author> <dateyear="2022" />month="September" year="2022"/> </front><seriesInfo name="Microsoft Blog" value="https://techcommunity.microsoft.com/t5/networking-blog/ledbat-background-data-transfer-for-windows/ba-p/3639278" /> <refcontent></refcontent><refcontent>Microsoft Networking Blog</refcontent> </reference> <reference anchor="Apple">target="https://developer.apple.com/videos/play/wwdc2021/10239/"> <front> <title>Reduce network delays for your app</title> <authorinitials="S.C." surname="Stuart" fullname="Cheshire"> <organization />initials="S" surname="Cheshire" fullname="Stuart Cheshire"> <organization/> </author> <authorinitials="V.G." surname="Vidhi" fullname="initials="V" surname="Goel" fullname="Vidhi Goel "><organization /><organization/> </author> <dateyear="2021" />year="2021"/> </front><seriesInfo name="WWDC21" value="https://developer.apple.com/videos/play/wwdc2021/10239/" /> <refcontent></refcontent><refcontent>Apple Worldwide Developers Conference (WWDC2021), Video</refcontent> </reference> <referenceanchor="COMNET3" >anchor="COMNET3"> <front> <title> Design, implementation and validation of a receiver-driven less-than-best-effort transport </title> <authorinitials="M.B."initials="M" surname="Bagnulo" fullname="Marcelo Bagnulo"><organization /><organization/> </author> <authorinitials="A.G." surname="Garcia-Martinez"initials="A" surname="García-Martínez" fullname="AlbertoGarcia-Martinez"> <organization />García-Martínez"> <organization/> </author> <author initials="A.M." surname="Mandalari" fullname="Anna Maria Mandalari"><organization /><organization/> </author> <authorinitials="P.B,"initials="P" surname="Balasubramanian" fullname="Praveen Balasubramanian"><organization /><organization/> </author> <authorinitials="D.H."initials="D" surname="Havey" fullname="Daniel Havey"><organization /><organization/> </author> <authorinitials="G.M."initials="G" surname="Montenegro" fullname="Gabriel Montenegro"><organization /><organization/> </author> <dateyear="2022" />month="September" year="2023"/> </front> <refcontent>Computer Networks, vol. 233</refcontent> <seriesInfoname="Computer Networks" value="Volume 233" /> <refcontent></refcontent>name="DOI" value="10.1016/j.comnet.2023.109841"/> </reference> <referenceanchor="COMNET2" >anchor="COMNET2"> <front> <title>When less is more: BBR versus LEDBAT++</title> <authorinitials="M.B."initials="M" surname="Bagnulo" fullname="Marcelo Bagnulo"><organization /><organization/> </author> <authorinitials="A.G." surname="Garcia-Martinez"initials="A" surname="García-Martínez" fullname="AlbertoGarcia-Martinez"> <organization />García-Martínez"> <organization/> </author> <dateyear="2022" />month="December" year="2022"/> </front> <refcontent>Computer Networks, vol. 219</refcontent> <seriesInfoname="Computer Networks" value="Volume 219" /> <refcontent></refcontent>name="DOI" value="10.1016/j.comnet.2022.109460"/> </reference> <referenceanchor="COMNET1" >anchor="COMNET1"> <front> <title>An experimental evaluation of LEDBAT++ </title> <authorinitials="M.B."initials="M" surname="Bagnulo" fullname="Marcelo Bagnulo"><organization /><organization/> </author> <authorinitials="A.G." surname="Garcia-Martinez"initials="A" surname="García-Martínez" fullname="AlbertoGarcia-Martinez"> <organization />García-Martínez"> <organization/> </author> <dateyear="2022" />month="July" year="2022"/> </front> <refcontent>Computer Networks, vol. 212</refcontent> <seriesInfoname='Computer Networks' value="Volume 212"/> <refcontent></refcontent>name="DOI" value="10.1016/j.comnet.2022.109036"/> </reference></references> <section title="Terminology"> <t>We use the following abreviations thoughout the text.<!-- [rfced] References: Weinclude a short listfound and added DOIs forthe reader's convenence:</t> <t><list> <t>RCV.WND: the value included in the Receive Window field of the TCP header (which computation is modified by this specification)</t> <t>SND.WND:[COMNET1], [COMNET2], and [COMNET3]. TheTCP sender's window</t> <t>cwnd: the consgestion window as computed by the congestion control algorithm running at the TCP sender.</t> <t>RLWND: the window value calculated by rLEDBAT algorithm</t> <t>fcwnd: the value that a standard RFC793bis TCP receiver calculates to set in the receive window for flow control purposes.</t> <t>RCV.HGH: the highest sequence number correspondingDOIs lead toa received byteopen-access versions ofdata at one point in time</t> <t>TSV.HGH: TSV.HGH the TSVal value corresponding to the segment in which RCV.HGH was carried at that point in time</t> <t>SEG.SEQ:those references. Please review our updates and thesequence numbernew links, and let us know if anything is incorrect. Original: [COMNET1] Bagnulo, M.B. and A.G. Garcia-Martinez, "An experimental evaluation ofthe last received segment</t> <t>TSV.SEQ: the TSVal valueLEDBAT++", Computer Networks Volume 212, 2022. [COMNET2] Bagnulo, M.B. and A.G. Garcia-Martinez, "When less is more: BBR versus LEDBAT++", Computer Networks Volume 219, 2022. [COMNET3] Bagnulo, M.B., Garcia-Martinez, A.G., Mandalari, A.M., Balasubramanian, P.B,., Havey, D.H., and G.M. Montenegro, "Design, implementation and validation ofthe last received segment</t> </list></t> </section>a receiver- driven less-than-best-effort transport", Computer Networks Volume 233, 2022. Currently: [COMNET1] Bagnulo, M. and A. García-Martínez, "An experimental evaluation of LEDBAT++", Computer Networks, vol. 212, DOI 10.1016/j.comnet.2022.109036, July 2022, <https://doi.org/10.1016/j.comnet.2022.109036>. [COMNET2] Bagnulo, M. and A. García-Martínez, "When less is more: BBR versus LEDBAT++", Computer Networks, vol. 219, DOI 10.1016/j.comnet.2022.109460, December 2022, <https://doi.org/10.1016/j.comnet.2022.109460>. [COMNET3] Bagnulo, M., García-Martínez, A., Mandalari, A.M., Balasubramanian, P., Havey, D., and G. Montenegro, "Design, implementation and validation of a receiver- driven less-than-best-effort transport", Computer Networks, vol. 233, DOI 10.1016/j.comnet.2023.109841, September 2023, <https://doi.org/10.1016/j.comnet.2023.109841>. --> </references> </references> <sectiontitle="rLEDBAT pseudo-code"> <t>We nextnumbered="true" toc="default"> <name>rLEDBAT Pseudocode</name> <t>In this section, we describe how to integrate the proposed rLEDBAT mechanisms and an LBE delay-based congestion control algorithm such as LEDBAT or LEDBAT++.We We describe the integrated algorithm as twoprocedures,procedures: one that is executed when a packet is received by a rLEDBAT-enabled endpoint(Figure 2)(<xref target="fig2"/>) and another that is executed when the rLEDBAT-enabled endpoint sends a packet(Figure 3).(<xref target="fig3"/>). At the beginning, RLWND is set to its maximum value, so that the sending rate of the sender is governed by the flow control algorithm of the receiver and the TCP slow start mechanism of the sender, and the ackedBytes variable is set to 0. </t> <t>We assume that the LBE congestion control algorithm defines a WindowIncrease() function and a WindowDecrease() function. For example, in the case of LEDBAT++, the WindowIncrease() function is an additive increase, while the WindowDecrease() function is a multiplicative decrease. In the case of theWindowIncrease(),WindowIncrease() function, we assume that it takes as input the current window size and the number of bytes that were acknowledged since the last window update (ackedBytes) and returns as output the updated window size. In the case ofWindowDecrease(),the WindowDecrease() function, it takes as input the current window size and returns the updated window size. </t> <t>The data structures used in the algorithms are as follows. The sentList is a list that contains the TSval and the local send time of each packet sent by the rLEDBAT-enabled endpoint. The TSecr field of the packets received by the rLEDBAT-enabled endpoint is matched with the sendList to compute the RTT. <!-- [rfced] Appendix B: As it appears that "TSecr field" should remain singular (i.e., not be "TSecr fields") and "TSecr field" is the subject of this sentence, we changed "are" to "is". Please let us know if "TSecr field" should be "TSecr fields" instead. Original: The TSecr field of the packets received by the rLEDBAT-enabled endpoint are matched with the sendList to compute theRTT.</t>RTT. Currently: The TSecr field of the packets received by the rLEDBAT-enabled endpoint is matched with the sendList to compute the RTT. --> </t> <t>The RTT values computed for each received packet are stored in the RTTlist, whichcontainsalso contains the received TSecr (to avoid using multiple packets with the same TSecr for RTT calculations, only the first packet received for a given TSecr is used to compute the RTT). It also contains the local time at which the packet was received, to allow selecting the RTTs measured in a given period (e.g., in the last 10 minutes). RTTlist is initialized with all its values to its maximum.</t> <figuretitle="Procedure executed whenanchor="fig2"> <name>Procedure Executed When apacket is received"> <sourcecode>Packet Is Received</name> <sourcecode type="pseudocode"><![CDATA[ procedure receivePacket() //Looks for first sent packet with same TSval as TSecr,and,and //returns time difference receivedRTT = computeRTT(sentList, receivedTSecr, receivedTime) //Inserts minimum value for a given receivedTSecr//note//Note that many received packets may contain same receivedTSecr insertRTT (RTTlist, receivedRTT, receivedTSecr, receivedTime) filteredRTT = minLastKMeasures(RTTlist, K=4) baseRTT = minLastNSeconds(RTTlist, N=180) qd = filteredRTT - baseRTT //ackedBytes is the number of bytes that can be used to reduce //the Receive Window - without shrinking it - if necessary ackedBytes = ackedBytes + receiveBytes if retransmittedPacketDetected then RLWND = DecreaseWindow(RLWND)// Only//Only once per RTT end if if qd<< T then RLWND = IncreaseWindow(RLWND, ackedBytes) else RLWND = DecreaseWindow(RLWND) end if end procedure</sourcecode>]]></sourcecode> </figure> <!-- [rfced] Figures 2 and 3: Per the contents of the figures and the title of Appendix B, we set the sourcecode type to "pseudocode". Please let us know any concerns. Please see <https://www.rfc-editor.org/rpc/wiki/doku.php?id=sourcecode-types> for a list of sourcecode types. --> <figuretitle="Procedure executed whenanchor="fig3"> <name>Procedure Executed When apacket is sent"> <sourcecode>Packet Is Sent</name> <sourcecode type="pseudocode"><![CDATA[ procedure SENDPACKET if (RLWND > RLWNDPrevious) or (RLWND - RLWNDPrevious<< ackedBytes) then RLWNDPrevious = RLWND else RLWNDPrevious = RLWND - ackedBytes end if ackedBytes = 0 RLWNDPrevious = RLWND //Compute the RWND to include in the packet RLWND = min(RLWND, fcwnd) end procedure</sourcecode>]]></sourcecode> </figure> <!-- [rfced] Figure 3: Should "RWND" be "RLWND" here? We ask because we do not see "RWND" used elsewhere in this document. Original: //Compute the RWND to include in the packet RLWND = min(RLWND, fcwnd) --> </section> <section numbered="false" toc="default"> <name>Acknowledgments</name> <t>This work was supported by the EU through the StandICT projects RXQ, CCI, and CEL6; the NGI Pointer RIM project; and the H2020 5G-RANGE project; and by the Spanish Ministry of Economy and Competitiveness through the 5G-City project (TEC2016-76795-C6-3-R).</t> <t>We would like to thank ICCRG chairs <contact fullname="Reese Enghardt"/> and <contact fullname="Vidhi Goel"/> for their support on this work. We would also like to thank <contact fullname="Daniel Havey"/> for his help. We would like to thank <contact fullname="Colin Perkins"/>, <contact fullname="Mirja Kühlewind"/>, and <contact fullname="Vidhi Goel"/> for their reviews and comments on earlier draft versions of this document.</t> </section> </back> <!-- [rfced] FYI - We have added expansions for the following abbreviations per Section 3.6 of RFC 7322 ("RFC Style Guide"). Please review each expansion in the document carefully to ensure correctness. Controlled Delay (CoDel) Proportional Integral controller Enhanced (PIE) Low Latency, Low Loss, and Scalable Throughput (L4S) Maximum Segment Size (MSS) Bottleneck Bandwidth and Round-trip propagation time (BBR) --> <!-- [rfced] Please review the "Inclusive Language" portion of the online Style Guide at <https://www.rfc-editor.org/styleguide/part2/#inclusive_language>, and let us know if any changes are needed. Updates of this nature typically result in more precise language, which is helpful for readers. Note that our script did not flag any words in particular, but this should still be reviewed as a best practice. --> <!-- [rfced] Please let us know if any changes are needed for the following: a) The following terms were used inconsistently in this document. We chose to use the latter forms. Please let us know any objections. Congestion control (1 instance) / congestion control (46 instances) RCV-WND (Figure 1) / RCV WND (Section 5) / RCV.WND (per the rest of this document and per published RFCs to date) TSVal / TSval (per published RFCs, including RFC 7323; we could not find "TSVal" in any published RFC) b) The following terms appear to be used inconsistently in this document. Please let us know which form is preferred. a rLEDBAT / an rLEDBAT Receive window / Receive Window / receive window (We see that "congestion window" is used consistently.) sendList / sentList --> </rfc>