Engineering blog
November 19, 2026 Michael Freeman
Longshore: A 36 GB/s Zero-Allocation I/O and Object Storage Layer in Rust
Longshore is the Rust I/O library under ServiceRadar's video storage and UltraGraphDB. It runs one io_uring ring per core, and one thread uploads to Amazon S3 at 36.4 GB/s from a 300 Gbit/s Graviton4 instance.
Off-the-shelf cloud storage SDKs make blocking DNS calls, run hidden thread pools,
allocate on the heap for every request, and copy buffers more often than they need to.
That leaves an order of magnitude of NIC and NVMe throughput unused.
Longshore
is about 25,000 lines of Rust that we wrote to get that throughput back. It runs one
io_uring
ring per core on a plain thread instead of an async runtime, takes no locks on the data
path, and does all of its heap allocation at startup.
Against Amazon S3, one Longshore thread on a Graviton4 instance with a 300 Gbit/s card uploads at 36.4 GB/s, or 16.7 GB/s with a CRC-64/NVME checksum on every body. Plain GET is 7.9 GB/s on an AMD Zen 5 instance. The same ring-per-core boundary also carries direct-I/O NVMe linked chains, a deterministic power-loss disk simulator, registered-ring server networking, and lock-free mailboxes between cores, so a storage service can use it as its only I/O layer.
Design rules
Each core runs one thread, and no other thread touches that core's ring, buffers, or file slots. Every subsystem in Longshore follows the same four rules:
1. One thread per core
A single thread handles all of a core's sockets, files, timers, and cross-core wakeups from one completion queue, so the data path needs neither locks nor an async runtime.
2. Memory is allocated once
Buffers, file slots, socket tables, and mailbox rings are allocated once when the
Io
is built. After that,
alloc_guard::Guard
seals the heap and panics if anything allocates.
3. Protocols are state machines
The HTTP, DNS, TLS, S3, and consumer code submits
Op
values tagged with 64-bit tokens and handles the
Completion
results. All I/O goes through the ring.
4. Cores talk through mailboxes
Each core keeps its own rings, buffers, and file slots. When cores need to talk, one
sends the other a bounded, fixed-size
Copy
message through longshore_mailbox.
io_uring: one thread, two rings shared with the kernel
Socket, file, and timer ops enter the submission queue (SQ) in one batch; completions return on the completion queue (CQ). Linux 6.1+.
The subsystems
Portable backend
Engine / World split
A single Engine<W>
implements the I/O contract (timers, cancellation, backpressure) over either a
virtual world for seeded simulation or a POSIX world (kqueue/epoll). It passes the same contract test suite as the Linux
UringIo
backend.
File operations
Direct I/O & linked chains
Fixed-size FileSlot
tables,
block-aligned registered buffer pools with upfront
RLIMIT_MEMLOCK
checks, and submit_chain
(IOSQE_IO_LINK) let a WAL write
and fdatasync
enter the kernel once.
Simulated disk
Power loss & 11 faults
SimDisk separates durable storage
from an unsynced write-log with whole-machine power_loss, 4096-byte sector-cut
tearing, causal directory durability, and 11 seeded or placed disk faults (including
writeback AS_EIO poisoning and
bit flips).
Server networking
Armed rings & optional TLS
Multi-completion AcceptArmed
and RecvArmed
share a registered buffer ring, so an idle plain connection holds no buffers. The
server side also has CQ NODROP
sizing, Unix-domain sockets, and longshore_stream,
where TLS is an optional Cargo feature.
Inter-core mailbox
Lock-free MPSC queue & S3 client
longshore_mailbox
passes bounded, fixed-size Copy
messages directly into a core's
Io
wakeup path. Future
and blocking callers,
longshore_s3::Client
among them, reach a dedicated I/O ring through it without taking a lock. 27
loom
concurrency models verify it.
Longshore inter-core mailbox architecture
Callers send fixed-size Copy messages into a bounded lock-free MPSC queue; one core thread owns the ring.
Benchmarks: live Amazon S3, NVMe chains, and Loom
1. Amazon S3 next to other clients (1 thread, 8 MiB objects)
We measured Longshore from two instances in us-east-1, both on Linux 7.0: a
c8gn.24xlarge
(Graviton4, 300 Gbit/s card) and an
m8azn.24xlarge
(AMD Zen 5, 200 Gbit/s card). Each figure is the median of five 8-second runs with one
client thread. The other rows are the published numbers for aws-sdk-cpp, aws-sdk-go-v2, anyblob, and TigerBeetle's Zig client from their benchmark talk.
On Graviton4, Longshore is ahead of TigerBeetle's figures on plain PUT and CRC-64/NVME PUT and slightly behind on GET. On the AMD instance it is ahead on GET and CRC-64/NVME PUT and behind on plain PUT. Plain PUT on Graviton4 fills the 300 Gbit/s card with about half of the client thread idle. That counts Longshore's thread only. Across the whole machine about 3.3 cores are busy during that run, and the other 2.8 are kernel network work on other cores. The talk reports its figure per core and does not say whether it counts that work.
TigerBeetle's 4.3 GB/s in the checksum column is the speed of its CRC-64 on its own.
Longshore computes CRC-64/NVME with its own kernels, which use carry-less multiplication
(PMULL on Arm, VPCLMULQDQ on x86).
Measured alone on one Cortex-X925 core of the DGX Spark, over 8 MiB buffers with a warm
cache, they run at 88 GB/s
(median of three runs, 86.2 to 88.6). That is a best case, because inside a PUT each
8 MiB buffer is cold in the cache.
| Client | Plain PUT | PUT + CRC-64 | Plain GET |
|---|---|---|---|
| aws-sdk-cpp | 1.7 GB/s | — | 1.3 GB/s |
| aws-sdk-go-v2 | 1.9 GB/s | — | 1.7 GB/s |
| anyblob | 5.3 GB/s | — | 3.8 GB/s |
| TigerBeetle (Zig) | 25.2 GB/s | 4.3 GB/s | 4.7 GB/s |
| Longshore, Graviton4 | 36.4 GB/s | 16.7 GB/s | 4.6 GB/s |
| Longshore, AMD Zen 5 | 23.9 GB/s | 14.1 GB/s | 7.9 GB/s |
The Longshore figures need a setting that is off by default. S3's endpoint name resolves to several addresses. Some of them negotiate 8,228-byte TCP segments and others 1,432-byte ones, and neither instance's card segments in hardware, so the small ones cost the client thread several times more per byte. With the setting on, the client prefers the large-segment addresses. With it off, a run's throughput depends on which addresses its connections happen to land on.
On a smaller c8gn.8xlarge
(Graviton4, Linux 7.0.0-1017-aws, us-east-1)
the 100 Gbit/s card is the limit. Plain PUT fills it while using
17% of one CPU core, which works out to 72.3 GB per CPU-second, and HTTPS PUT holds
3.94 GB/s. These are medians of three runs. None of the 18 runs had to retry a request, and the
correctness suites on this instance passed with allocation_sealed=true, meaning no heap allocation after startup.
The cores column in the table counts the client thread only.
| Workload | Median (range) | Cores used | GB per CPU-second |
|---|---|---|---|
| HTTP PUT (192 conns) | 12.14 GB/s (12.07–12.24) | 0.17 | 72.3 |
| HTTP PUT + CRC-64 (192 conns) | 11.83 GB/s (10.25–12.25) | 0.82 | 14.5 |
| HTTP GET (64 conns) | 3.96 GB/s (3.81–4.13) | 0.99 | 4.0 |
| HTTPS PUT (64 conns) | 3.94 GB/s (3.80–3.97) | 0.93 | 4.2 |
| HTTPS PUT + CRC-64 (64 conns) | 3.67 GB/s (3.65–3.78) | 0.97 | 3.8 |
| HTTPS GET (64 conns) | 2.53 GB/s (2.41–2.62) | 0.97 | 2.6 |
2. Direct-I/O linked chains (file_uring)
The WAL-style test is a 4 KiB
O_DIRECT
overwrite into a preallocated file followed by IORING_FSYNC_DATASYNC. Submitting both as one
IOSQE_IO_LINK
chain removes the kernel round trip between the write completing and the sync being
submitted. The chain was faster at every percentile on the DGX Spark. On the Xeon it
was faster at p50 and p95 and slower at p99 and on the mean.
| Host / storage | Mode | p50 | p95 | p99 | Mean |
|---|---|---|---|---|---|
| DGX Spark (aarch64, Linux 7.0, ext4 NVMe) | two_ops | 1,892.8 µs | 5,163.3 µs | 6,517.3 µs | 2,354.3 µs |
| DGX Spark (aarch64, Linux 7.0, ext4 NVMe) | chain (IOSQE_IO_LINK) | 1,450.5 µs (-23.4%) | 2,424.9 µs (-53.0%) | 3,315.4 µs (-49.1%) | 1,561.8 µs (-33.7%) |
| Xeon 8358P (x86_64, Linux 6.8, XFS) | two_ops | 303.8 µs | 928.6 µs | 1,197.6 µs | 428.6 µs |
| Xeon 8358P (x86_64, Linux 6.8, XFS) | chain (IOSQE_IO_LINK) | 277.8 µs (-8.6%) | 910.3 µs (-2.0%) | 3,410.8 µs | 443.8 µs |
3. Exhaustive concurrency verification (loom)
longshore_s3::Client is built on longshore_mailbox,
which keeps the state space that loom has to explore small. The 10 client_model
tests cover up to 20,183 interleavings and finish in 3.50 seconds,
and the 17 mailbox_model tests take 0.35 seconds.
Where the CPU time goes in TLS
In loopback benchmarks on the DGX Spark (an
aarch64
system with 10 Cortex-A725 efficiency cores and 10 Cortex-X925 performance cores running
Linux 7.0), plain HTTP GET reaches
11.18 GB/s
on one core and plain PUT reaches 5.70 GB/s. Over TLS, PUT manages 3.6 GB per CPU-second and GET 4.4. We wanted to know whether Longshore's Rust
framing or
io_uring
layer was holding TLS back, or whether it was the cipher.
We profiled
http_uring
on the DGX Spark with
perf record -F 999 -g --call-graph fp
(client pinned to Cortex-X925 CPU 19, loopback sink pinned to Cortex-X925 CPU 18 so
big.LITTLE scheduling couldn't skew the sink). Longshore's Rust userspace code is under 1% of CPU.
About half of the core (56.87% on PUT, 49.62% on GET) goes to AWS-LC's hand-written
ARMv8 AES-GCM kernels, and nearly all of the rest is Linux kernel TCP, netfilter, and __arch_copy_to_user.
perf
flamegraph of TLS PUT
on the DGX Spark
(5,082 Cortex-X925 samples,
4.56 GB/CPU-s). 56.87%
is inside
aws_lc_0_47_0_aes_gcm_enc_kernel
(red), with the remainder in kernel TCP/skb/netfilter (yellow-orange).
perf
flamegraph of TLS GET
on the DGX Spark
(6,741 Cortex-X925 samples,
3.93 GB/CPU-s). 49.62%
is inside
aws_lc_0_47_0_aes_gcm_dec_kernel
(red) and 11.26%
is __arch_copy_to_user
in tcp_recvmsg.
With Longshore's own code under 1% of the profile, getting single-core TLS much past ~4.5 GB/s on this hardware depends on faster hardware-accelerated AEAD kernels or kernel/NIC offload.
What we use Longshore for
Longshore is the I/O and storage layer under two ServiceRadar workloads:
ServiceRadar NVR / VMS storage
Video ingestion and object-storage tiering need predictable tail latency and line-rate uploads, which rules out GC pauses and allocator churn. With Longshore's zero-copy S3 client and direct-I/O file rings, edge and core collectors stream continuous media segments to local NVMe and S3-compatible object stores on a fraction of a core.
UltraGraphDB
Our distributed graph database for ServiceRadar's topology engine uses Longshore's ring-per-core NVMe linked chains, armed server networking, inter-core mailboxes, and deterministic power-loss simulation. Every storage and replication path is tested under fault injection before it runs on production hardware.
Longshore is open source under the Apache 2.0 license at github.com/carverauto/longshore.