Skip to content

ServiceRadar

Open navigation

Engineering blog

November 19, 2026 Michael Freeman

Longshore: A 36 GB/s Zero-Allocation I/O and Object Storage Layer in Rust

Longshore is the Rust I/O library under ServiceRadar's video storage and UltraGraphDB. It runs one io_uring ring per core, and one thread uploads to Amazon S3 at 36.4 GB/s from a 300 Gbit/s Graviton4 instance.

longshore io-uring rust storage simulation s3 performance tls

Off-the-shelf cloud storage SDKs make blocking DNS calls, run hidden thread pools, allocate on the heap for every request, and copy buffers more often than they need to. That leaves an order of magnitude of NIC and NVMe throughput unused. Longshore is about 25,000 lines of Rust that we wrote to get that throughput back. It runs one io_uring ring per core on a plain thread instead of an async runtime, takes no locks on the data path, and does all of its heap allocation at startup.

Against Amazon S3, one Longshore thread on a Graviton4 instance with a 300 Gbit/s card uploads at 36.4 GB/s, or 16.7 GB/s with a CRC-64/NVME checksum on every body. Plain GET is 7.9 GB/s on an AMD Zen 5 instance. The same ring-per-core boundary also carries direct-I/O NVMe linked chains, a deterministic power-loss disk simulator, registered-ring server networking, and lock-free mailboxes between cores, so a storage service can use it as its only I/O layer.

Design rules

Each core runs one thread, and no other thread touches that core's ring, buffers, or file slots. Every subsystem in Longshore follows the same four rules:

1. One thread per core

A single thread handles all of a core's sockets, files, timers, and cross-core wakeups from one completion queue, so the data path needs neither locks nor an async runtime.

2. Memory is allocated once

Buffers, file slots, socket tables, and mailbox rings are allocated once when the Io is built. After that, alloc_guard::Guard seals the heap and panics if anything allocates.

3. Protocols are state machines

The HTTP, DNS, TLS, S3, and consumer code submits Op values tagged with 64-bit tokens and handles the Completion results. All I/O goes through the ring.

4. Cores talk through mailboxes

Each core keeps its own rings, buffers, and file slots. When cores need to talk, one sends the other a bounded, fixed-size Copy message through longshore_mailbox.

io_uring: one thread, two rings shared with the kernel

Socket, file, and timer ops enter the submission queue (SQ) in one batch; completions return on the completion queue (CQ). Linux 6.1+.

USER KERNEL HW Core thread Registered pool pinned at build sealed() = 0 allocs submit() SQ CQ NVMe SSD · 100G NIC INVARIANT 1 No copies or allocations registered buf IDs only INVARIANT 2 One syscall per turn SINGLE_ISSUER | DEFER

The subsystems

Portable backend

Engine / World split

A single Engine<W> implements the I/O contract (timers, cancellation, backpressure) over either a virtual world for seeded simulation or a POSIX world (kqueue/epoll). It passes the same contract test suite as the Linux UringIo backend.

File operations

Direct I/O & linked chains

Fixed-size FileSlot tables, block-aligned registered buffer pools with upfront RLIMIT_MEMLOCK checks, and submit_chain (IOSQE_IO_LINK) let a WAL write and fdatasync enter the kernel once.

Simulated disk

Power loss & 11 faults

SimDisk separates durable storage from an unsynced write-log with whole-machine power_loss, 4096-byte sector-cut tearing, causal directory durability, and 11 seeded or placed disk faults (including writeback AS_EIO poisoning and bit flips).

Server networking

Armed rings & optional TLS

Multi-completion AcceptArmed and RecvArmed share a registered buffer ring, so an idle plain connection holds no buffers. The server side also has CQ NODROP sizing, Unix-domain sockets, and longshore_stream, where TLS is an optional Cargo feature.

Inter-core mailbox

Lock-free MPSC queue & S3 client

longshore_mailbox passes bounded, fixed-size Copy messages directly into a core's Io wakeup path. Future and blocking callers, longshore_s3::Client among them, reach a dedicated I/O ring through it without taking a lock. 27 loom concurrency models verify it.

Longshore inter-core mailbox architecture

Callers send fixed-size Copy messages into a bounded lock-free MPSC queue; one core thread owns the ring.

USER KERNEL HW longshore_mailbox (bounded MPSC · Copy msgs) Caller 1 Future waiter slot Caller 2 block_on waiter slot Core B peer core waiter slot Core C peer core waiter slot Core I/O worker one thread · 27 loom SQ CQ NVMe SSDs · Sockets

Benchmarks: live Amazon S3, NVMe chains, and Loom

1. Amazon S3 next to other clients (1 thread, 8 MiB objects)

We measured Longshore from two instances in us-east-1, both on Linux 7.0: a c8gn.24xlarge (Graviton4, 300 Gbit/s card) and an m8azn.24xlarge (AMD Zen 5, 200 Gbit/s card). Each figure is the median of five 8-second runs with one client thread. The other rows are the published numbers for aws-sdk-cpp, aws-sdk-go-v2, anyblob, and TigerBeetle's Zig client from their benchmark talk.

On Graviton4, Longshore is ahead of TigerBeetle's figures on plain PUT and CRC-64/NVME PUT and slightly behind on GET. On the AMD instance it is ahead on GET and CRC-64/NVME PUT and behind on plain PUT. Plain PUT on Graviton4 fills the 300 Gbit/s card with about half of the client thread idle. That counts Longshore's thread only. Across the whole machine about 3.3 cores are busy during that run, and the other 2.8 are kernel network work on other cores. The talk reports its figure per core and does not say whether it counts that work.

TigerBeetle's 4.3 GB/s in the checksum column is the speed of its CRC-64 on its own. Longshore computes CRC-64/NVME with its own kernels, which use carry-less multiplication (PMULL on Arm, VPCLMULQDQ on x86). Measured alone on one Cortex-X925 core of the DGX Spark, over 8 MiB buffers with a warm cache, they run at 88 GB/s (median of three runs, 86.2 to 88.6). That is a best case, because inside a PUT each 8 MiB buffer is cold in the cache.

Client Plain PUT PUT + CRC-64 Plain GET
aws-sdk-cpp 1.7 GB/s — 1.3 GB/s
aws-sdk-go-v2 1.9 GB/s — 1.7 GB/s
anyblob 5.3 GB/s — 3.8 GB/s
TigerBeetle (Zig) 25.2 GB/s 4.3 GB/s 4.7 GB/s
Longshore, Graviton4 36.4 GB/s 16.7 GB/s 4.6 GB/s
Longshore, AMD Zen 5 23.9 GB/s 14.1 GB/s 7.9 GB/s

The Longshore figures need a setting that is off by default. S3's endpoint name resolves to several addresses. Some of them negotiate 8,228-byte TCP segments and others 1,432-byte ones, and neither instance's card segments in hardware, so the small ones cost the client thread several times more per byte. With the setting on, the client prefers the large-segment addresses. With it off, a run's throughput depends on which addresses its connections happen to land on.

On a smaller c8gn.8xlarge (Graviton4, Linux 7.0.0-1017-aws, us-east-1) the 100 Gbit/s card is the limit. Plain PUT fills it while using 17% of one CPU core, which works out to 72.3 GB per CPU-second, and HTTPS PUT holds 3.94 GB/s. These are medians of three runs. None of the 18 runs had to retry a request, and the correctness suites on this instance passed with allocation_sealed=true, meaning no heap allocation after startup. The cores column in the table counts the client thread only.

Workload Median (range) Cores used GB per CPU-second
HTTP PUT (192 conns) 12.14 GB/s (12.07–12.24) 0.17 72.3
HTTP PUT + CRC-64 (192 conns) 11.83 GB/s (10.25–12.25) 0.82 14.5
HTTP GET (64 conns) 3.96 GB/s (3.81–4.13) 0.99 4.0
HTTPS PUT (64 conns) 3.94 GB/s (3.80–3.97) 0.93 4.2
HTTPS PUT + CRC-64 (64 conns) 3.67 GB/s (3.65–3.78) 0.97 3.8
HTTPS GET (64 conns) 2.53 GB/s (2.41–2.62) 0.97 2.6

2. Direct-I/O linked chains (file_uring)

The WAL-style test is a 4 KiB O_DIRECT overwrite into a preallocated file followed by IORING_FSYNC_DATASYNC. Submitting both as one IOSQE_IO_LINK chain removes the kernel round trip between the write completing and the sync being submitted. The chain was faster at every percentile on the DGX Spark. On the Xeon it was faster at p50 and p95 and slower at p99 and on the mean.

Host / storage Mode p50 p95 p99 Mean
DGX Spark (aarch64, Linux 7.0, ext4 NVMe) two_ops 1,892.8 µs 5,163.3 µs 6,517.3 µs 2,354.3 µs
DGX Spark (aarch64, Linux 7.0, ext4 NVMe) chain (IOSQE_IO_LINK) 1,450.5 µs (-23.4%) 2,424.9 µs (-53.0%) 3,315.4 µs (-49.1%) 1,561.8 µs (-33.7%)
Xeon 8358P (x86_64, Linux 6.8, XFS) two_ops 303.8 µs 928.6 µs 1,197.6 µs 428.6 µs
Xeon 8358P (x86_64, Linux 6.8, XFS) chain (IOSQE_IO_LINK) 277.8 µs (-8.6%) 910.3 µs (-2.0%) 3,410.8 µs 443.8 µs

3. Exhaustive concurrency verification (loom)

longshore_s3::Client is built on longshore_mailbox, which keeps the state space that loom has to explore small. The 10 client_model tests cover up to 20,183 interleavings and finish in 3.50 seconds, and the 17 mailbox_model tests take 0.35 seconds.

Where the CPU time goes in TLS

In loopback benchmarks on the DGX Spark (an aarch64 system with 10 Cortex-A725 efficiency cores and 10 Cortex-X925 performance cores running Linux 7.0), plain HTTP GET reaches 11.18 GB/s on one core and plain PUT reaches 5.70 GB/s. Over TLS, PUT manages 3.6 GB per CPU-second and GET 4.4. We wanted to know whether Longshore's Rust framing or io_uring layer was holding TLS back, or whether it was the cipher.

We profiled http_uring on the DGX Spark with perf record -F 999 -g --call-graph fp (client pinned to Cortex-X925 CPU 19, loopback sink pinned to Cortex-X925 CPU 18 so big.LITTLE scheduling couldn't skew the sink). Longshore's Rust userspace code is under 1% of CPU. About half of the core (56.87% on PUT, 49.62% on GET) goes to AWS-LC's hand-written ARMv8 AES-GCM kernels, and nearly all of the rest is Linux kernel TCP, netfilter, and __arch_copy_to_user.

Flamegraph of longshore http_uring TLS PUT on DGX Spark aarch64 Cortex-X925 showing 56.87% in aws_lc_0_47_0_aes_gcm_enc_kernel
Figure 1. perf flamegraph of TLS PUT on the DGX Spark (5,082 Cortex-X925 samples, 4.56 GB/CPU-s). 56.87% is inside aws_lc_0_47_0_aes_gcm_enc_kernel (red), with the remainder in kernel TCP/skb/netfilter (yellow-orange).
Flamegraph of longshore http_uring TLS GET on DGX Spark aarch64 Cortex-X925 showing 49.62% in aws_lc_0_47_0_aes_gcm_dec_kernel and 11.26% in __arch_copy_to_user
Figure 2. perf flamegraph of TLS GET on the DGX Spark (6,741 Cortex-X925 samples, 3.93 GB/CPU-s). 49.62% is inside aws_lc_0_47_0_aes_gcm_dec_kernel (red) and 11.26% is __arch_copy_to_user in tcp_recvmsg.

With Longshore's own code under 1% of the profile, getting single-core TLS much past ~4.5 GB/s on this hardware depends on faster hardware-accelerated AEAD kernels or kernel/NIC offload.

What we use Longshore for

Longshore is the I/O and storage layer under two ServiceRadar workloads:

ServiceRadar NVR / VMS storage

Video ingestion and object-storage tiering need predictable tail latency and line-rate uploads, which rules out GC pauses and allocator churn. With Longshore's zero-copy S3 client and direct-I/O file rings, edge and core collectors stream continuous media segments to local NVMe and S3-compatible object stores on a fraction of a core.

UltraGraphDB

Our distributed graph database for ServiceRadar's topology engine uses Longshore's ring-per-core NVMe linked chains, armed server networking, inter-core mailboxes, and deterministic power-loss simulation. Every storage and replication path is tested under fault injection before it runs on production hardware.

Longshore is open source under the Apache 2.0 license at github.com/carverauto/longshore.