July 28, 2026 · Engineering

The serving path, measured in microseconds

Nyx publishes two numbers about itself: 71µs of routing overhead at p50 and 235µs at p99. That is the time our infrastructure adds between accepting your request and handing it to a model server. Most latency talk in inference is about the model, tokens per second and time to first token, and those numbers belong mostly to the GPU. The serving path is the part we own outright, so it is the part we measure in public.

The measurement covers everything between the two sockets: TLS session reuse, API key verification, rate accounting, model lookup, backend selection, and the write to an already-open connection to the pod. It excludes GPU time and the network on either side, because neither is ours to promise.

Loopback methodology

The numbers come from loopback probes rather than sampled production traffic. In each region a prober sends about 2,000 requests per minute through the public endpoint, addressed to a stub model that acknowledges instantly and does no compute. Wall time minus the stub's own turnaround is the serving path. Probes travel the exact production path, auth and rate accounting included, so there is no separate fast lane to flatter the graph. The published p50 and p99 are computed over the trailing 24 hours and refreshed each minute at /health.

Streaming gets the same discipline. Responses pass through byte for byte: chunks leave our edge as the model server emits them, with no re-buffering and no coalescing, so the overhead you measured on the first byte holds for every byte after it. If a model emits a 3-byte chunk, your socket receives a 3-byte chunk.

The last policy is what happens at capacity. Every model has a declared capacity, and past it Nyx returns 429 immediately, with a Retry-After header, instead of queueing. A queue converts overload into latency, which is the one failure mode a published p99 cannot warn you about until it has already happened. An immediate 429 costs your client one retry or a short backoff; a silent 40-second queue costs the request its whole timeout budget.

{
  "status": "ok",
  "region": "us-east-1",
  "routing_overhead_us": { "p50": 71, "p99": 235 },
  "probe_window_s": 86400,
  "probe_rate_per_min": 2000,
  "declared_capacity_rps": 1200,
  "current_rps": 418,
  "accepting": true,
  "models_live": 10
}

That response is live and unauthenticated, and it is the same data behind the status line in our footer. If the p99 drifts, it drifts there first, in public, before anyone writes a postmortem. Prices sit next door at /v1/models under the same principle: the API is the source of truth, and the marketing site quotes it.