One of our recent publications in the SIGCOMM Computer Communication Review (CCR) journal, “The Packet Number Space Debate in Multipath QUIC”, has been selected to be part of the “Best of CCR” session at the upcoming ACM SIGCOMM 2023 conference. Let us introduce in this blog post the context in which that work relates and the impact it had regarding the standardisation effort of Multipath QUIC at the IETF.

While our first proposals to bring multipath into QUIC date back from 2017 (both as an ACM CoNEXT 2017 publication and as an IETF draft), it only gained interest from the network community since 2020 and the publication of two other alternate proposals, one being authored by Alibaba. However, the presence of multiple proposals actually delayed the start of the Multipath QUIC standardization effort, as most of them were very specific to their use-cases (some more adapted to address mobility, others to Quality of Experience scheduling).

To make progress, the authors of the different proposals agreed to identify the core elements that any Multipath QUIC implementation must include. This led to the writting of the common proposal in October 2021, which served as the basis of the adopted Multipath QUIC draft at the IETF. These three proposals actually share most of the multipath extension principles. The extension is negotiated using a QUIC transport parameter, multiple paths can only be used after the session handshake completes, different Connection IDs are used over different network paths, path creation is based on the QUIC path validation mechanism, and they all override the default connection migration one.

Despite all these similarities, there was a core design element of multipath QUIC where proposals have different approaches: how QUIC packets should be numbered. In single-path QUIC, once the handshake completes, all the remaining packets share a single application data packet number space. Packet numbers are monotonically increasing and never reused (i.e., packets are never retransmitted, but their content can be). The receiver can then acknowledge the reception of packets using ACK frames. Such ACK frames include ranges of received packets as well as the delay between the reception of the largest packet number and the transmission of the frame. This approach enables single-path QUIC to provide efficient and accurate loss recovery. On the one hand, one of the Multipath QUIC proposals suggests that this single packet number space can be shared among the different paths. For instance, packet N could be sent on one path and packet N+1 on another one. On the other hand, the two other proposals argue for path-dedicated, contiguous packet number spaces. Each packet number space would have its own identifier and each path could have a given packet N, where each path would have its own.

Packet numbering options

The above figure summarizes the difference between the Single Packet Number Space (SPNS) design and the Multiple Packet Number Spaces (MPNS) one. Each design has its pros and cons. SPNS, unlike MPNS, does not introduce any new frames on the wire (MPNS requires a new ACK_MP frame), does not change encryption internals (i.e., the nonce) and implementation experience reported less introduced code paths. On the other side, spreading packets numbers on different network paths having heterogeneous characteristics (bandwidth, delay,…) complexifies the recovery algorithm at the sender side and can make it hard for the sender to get accurate delay estimation of slow paths. SPNS also requires futher adaptation when, e.g., using ECN.

Yet, the SPNS design introduces transfer performance variability due to how the receiver maintains state regarding packets to acknowledge. In QUIC, receivers implement heuristics aiming at discarding old packet numbers from new acknowledgment reports and limiting the number of acknowledgment ranges. For instance, the picoquic implementation, one of the QUIC implementations supporting both numbering designs, limits the number of acknowledged ranges to 33. Furthermore, to keep bounded memory usage, many implementations integrate an ACK horizon mechanism where incoming packets with a number lower than a given threshold are considered as duplicate and therefore discarded. While picoquic implements a dynamic threshold depending on the perceived delay, such a threshold is often the value of the largest packet number seen minus a constant (127 being used in several implementations). In case of large out-of-order packet number reception, this may lead the receiver to not acknowledge packets (low ACK range limit), or even to discard valid packets (low ACK horizon).

To study such cases with high reordering at server side, we consider a bulk download of a 50 MB file over a single stream. We rely on picoquic, supporting both SPNS and MPNS variants (the paper also considers PQUIC with the multipath plugin that only supports MPNS, see the paper for details). We consider the NS3-DCE platform allowing to simulate network transfers with actual implementations, offering results with very low variability. We adopt an experimental design approach by considering 95 different 2-path configurations where the summed bandwidth of the paths is 100 Mbps and their summed RTT is 200 ms. Such an approach should provide network configurations where the expected completion time remains similar.

Completion time of 2-path configurations

The above figure show that while the performance of both designs with the default picoquic configuration are similar, we notice that SPNS has some performance decrease on configurations showing a large delay ratio between the paths (i.e., one path has a latency at least 4 times larger than the other). MPNS is less affected by such path delay asymmetry. If we now consider the receiver’s algorithms regarding ACK Range limitation (AR) and fixed ACK Horizon threshold (AH), we note that they strongly affect the performance of the traffic. While limiting the ACK range to 5 seems a bit extreme, several QUIC production implementations integrate a fixed ACK Horizon of 128. These lead to unacknowledged or dropped packets, implying spurious retransmissions and transfer stalls, hence limiting the transfer efficiency. On the other hand, the MPNS design is insensitive to these receiver’s heuristics as there is no out-of-order packet number reception due to the use of other paths.

While the effect of reordering is already visible on 2-path topologies, adding more heterogeneous paths would worsen the aforementioned issues. To assess this, we now consider 95 3-path configurations following the experimental design approach, where the summed path bandwidth is 100 Mbps and the summed path RTT is 300 ms.

Completion time of 3-path configurations

The above figure demonstrate that the picoquic implementation, with its internal limit of 33 ACK ranges, cannot rely on the SPNS design without strongly impacting the transfer performance. On the other hand, increasing this limit to 64 enables the SPNS design to match the MPNS performance.

In the end, both numbering designs can work, but the SPNS one introduces performance variability due to the receiver’s internals by design. Given Multipath QUIC may need to serve at some point much more complex network configurations and scenarios (e.g., spatial communications,…), such a property is undesirable. This is why this paper argues for considering the MPNS design for both performance and extension-composability (e.g., ECN, ACK frequency,…) purposes. With another study providing similar observations, this paper convinced the IETF to adopt the MPNS design in the standardized version of Multipath QUIC.

Note that the artifacts of the paper are available, enabling the community to reproduce the results presented in the paper.