How NuviraHub keeps a remote desktop responsive

In a remote desktop one number matters more than any other: the time between something changing on the remote screen and you seeing it. Above 100 ms the mouse feels like it is moving through honey; around 40 ms you stop noticing. This article explains how NuviraHub keeps that delay low and what went wrong along the way.

How we measure

Every video frame carries the moment it was captured. The viewer knows the offset between the two clocks, so it can compute the latency from capture to display of every frame, and every 250 ms it sends the median and the minimum back to the controlled PC. The same figures are shown live in the session bar and in the window title.

The numbers below come from a deliberately unfavourable set-up: a screen with a full-screen animation, so the image is always changing; a wired network. Quick Sync: controlled PC with an Intel Core i7-10710U, viewer on a laptop with an Intel Core i7-8550U. NVIDIA: controlled PC with an AMD Ryzen 7 5700X and an RTX 4060 Ti, viewer on the mini PC with the i7-10710U.

1. The image stays on the graphics card

The screen is captured with Windows' Desktop Duplication, which already hands over the image as a GPU texture. The GPU's video processor converts it to the format the encoder wants (NV12), and the hardware H.264 encoder takes it from there, through Media Foundation. The image never travels back to the processor's memory: the colour conversion takes about 1 ms instead of the 3–4 ms it takes on the CPU.

Not every PC has a usable hardware encoder, so at start-up NuviraHub tests the available ones and falls back to OpenH264 in software. Building OpenH264 with its assembly optimisations alone took it from 19 to 9 ms per frame.

2. Never let a frame get old

The first measurements between two real PCs were much worse than on a single machine. Three problems, all about frames waiting when they should not:

• A lock held while waiting. The capture call waited for a new image (up to 100 ms) while holding the lock of the shared graphics device, and conversion and encoding waited behind it: up to 360 ms for a single frame. Now capture polls without waiting.
• Images ageing in front of the encoder. A captured image waited for the encoder to be free, and was already old when encoded. Now the agent waits for the encoder first and only then takes the most recent image: latency dropped from 53 to 38 ms.
• A queue in the viewer. When the viewer was slower than the agent it still showed every frame in order, and latency grew up to 2.5 s. Now it always shows the last decoded frame and drops the ones already overtaken. On the sending side the network buffer is capped at 256 KB: beyond that the agent skips frames instead of queueing them.

3. Adapting to the network

On a clean local network a fixed bitrate is fine. On a network with delay and losses it is a disaster: with 25 ms, 1% loss and 5 Mbit/s the median latency was 2.2 seconds, because the video kept pushing more than the link could carry and everything piled up in queues.

Two changes fixed it. The first is an adaptive bitrate: when the median latency of the last second exceeds the minimum of the last ten seconds by 60 ms, for at least a second, the bitrate drops by 20%; when it stays close it rises by 15%, with a pause after each change. The second is BBR as QUIC's congestion control: with random losses, the classic algorithms (NewReno, Cubic) read every lost packet as congestion and used less than half of the bandwidth.

4. What the first test over a mobile network taught us

Simulations are tidy; a phone hotspot through the relay is not. The first real test over the Internet found five problems that no simulation had shown:

• Slow encoding mistaken for congestion. On a busy PC encoding took 130 ms per frame; the latency grew, and the adaptive bitrate cut the quality to the minimum, as if the network were at fault. Now it only looks at the network part: from the end of encoding to the arrival at the viewer.
• A still screen looking like a stuck network. With almost nothing to send, the latency of sparse packets on a mobile network jumps between 30 and 185 ms on its own, and the bitrate fell from 4 to 0.5 Mbit/s in 16 seconds with nothing moving. Now the bitrate changes only while the video is actually using a good part of it.
• Single peaks. Even with little traffic the mobile network had 150–600 ms spikes every few seconds. Reacting to each of them brought the bitrate to the minimum and photos became blurry. Only a queue lasting a whole second counts now.
• Blurry text on a still screen. The first frame after a change is encoded in a hurry, and on a slow link the text stayed unreadable. Now, when the screen stops changing, the last image is encoded again up to 15 times, about every 100 ms: each pass spends the bitrate on what the previous one left out, and once the image matches it costs almost nothing.
• An encoder that ignores changes. Through Media Foundation, Quick Sync ignores bitrate changes once a session has started. At first we restarted it, and the image stopped for about a second at every change; now the session stays open and the encoder is reconfigured with the new bitrate in 1–4 ms.

5. Direct when possible, and still encrypted when not

The two PCs try to reach each other directly through their routers (hole punching), also using local addresses when they are behind the same router. When that fails, the server's relay forwards the QUIC packets as they are, already encrypted end to end: in the local network the relay costs about 5 ms. On networks that block UDP the same packets travel inside a WebSocket, so the encryption and the session are identical; between our two test PCs this tunnel stayed at 38–39 ms.

NuviraHub is open source: the adaptive bitrate is in client/agent/src/abr.rs, the measurements and their history in the project documents. If you measure something on your network, especially with NVIDIA or AMD cards, we would like to hear about it: open an issue or write to support@nuvirahub.dev.