THE SERVING PATH
One path, measured end to end
Every layer a request crosses, from GPU kernels to the HTTP edge, is built in-house and tuned for one workload: streaming tokens from open-weight models.
Streamed
BYTE FOR BYTE, NO BUFFER
1M
LONGEST CONTEXT WINDOW
US-EAST-1
SERVING REGION
OPENAI
COMPATIBLE API
Nyx runs one serving path. A request enters at the HTTP edge, lands on a pod, prefills, and streams tokens back, and every layer it crosses on that trip is code we wrote and can change. Because the path is singular, it can be measured as a whole, and the measurement runs on the same route a customer request takes.
Measured like a customer
Loopback probes run the full serving path every minute. Each probe opens a fresh TCP connection, sends a real prompt to a real pod, and stops the clock at the first streamed byte. There is no instrumented shortcut and no internal timestamp doing the flattering work: the probe pays for DNS, the handshake, TLS, scheduling, and prefill, exactly as your client does.
What the probe reports is what the release process enforces. The probe percentiles are the gate: an engine build has to hold or improve them before it can take traffic, and a build that moves one the wrong way stays on the bench no matter what else it wins. The threshold lives in the release config rather than in a marketing sentence.
Refusing beats queueing
Past declared capacity, Nyx returns 429 immediately, with a Retry-After header that says when to come back. The alternative, an admission queue, looks kinder on a dashboard and is worse everywhere else: a queued request holds your connection open, burns your deadline, and usually dies as a timeout you cannot distinguish from an outage.
A refusal costs milliseconds and tells your retry logic the one thing it needs to know. It also keeps the measurement honest, because a percentile that quietly stops counting the requests that gave up is not measuring anything.
Every change earns its way in
The serving engine changes every week. Kernels get rewritten, cache layouts move, schedulers learn new tricks, and none of it touches customer traffic on the strength of a code review. Every candidate build replays recorded traces on shadow pods at production concurrency, and the replay decides: hold or improve the gated percentiles and the build ships, regress them and it dies at verify. The loop below is the real ledger. B-243, the prefix cache that carries across turns, went through it and now removes 31 percent of prefill on agent traces.