SOLUTIONS
Built for agent workloads
The prefix cache carries a session across turns, so each tool call pays prefill only on the tokens it has not sent before.
31%
PREFILL REMOVED ON AGENT TRACES
262K
TOKEN CONTEXT ON QWEN3-235B
OPENAI
COMPATIBLE API
1M
ON KIMI-K3, COMING SOON
Agents hit inference differently
An agent turn is not one big completion. It is a loop of short calls: the model picks a tool, the orchestrator runs it, the result lands in the transcript, and the whole thing goes back for the next step. Completions stay small while the prompt grows by a tool result every turn.
That shape inverts the usual cost. The same growing prefix, the system prompt, the tool schemas, the conversation, the accumulated tool output, is re-read on every call. Thirty turns in, the model spends most of its time re-reading a transcript it has already seen twenty-nine times.
What the serving path does about it
The engine's prefix cache carries across turns. When turn N+1 arrives with turn N's transcript as its prefix, the cached prefill is skipped and only the new tokens are computed. On agent traces that removes 31 percent of prefill work, and it shows up as faster first tokens deep in a session, right where loops usually bog down.
Sessions have room to grow: qwen3-235b-a22b-2507 holds 262,144 tokens of context and handles the tool-calling format natively, gpt-oss-120b runs the same loop, and kimi-k3, coming soon, raises the window to 1,000,000 tokens so file contents and tool output stay in the transcript instead of being summarized away. And past declared capacity Nyx returns 429 immediately with a Retry-After header, so an orchestrator can back off and retry deliberately instead of hanging inside a queue while its turn budget burns.
# Turn N of a tool loop: same prefix, one new tool result
curl https://api.nyxprovider.com/v1/chat/completions \
-H "Authorization: Bearer $NYX_API_KEY" \
-d '{
"model": "qwen3-235b-a22b-2507",
"stream": true,
"tools": [{"type": "function",
"function": {"name": "search_tickets"}}],
"messages": [
{"role": "user",
"content": "Which open tickets mention the billing bug?"},
{"role": "assistant",
"tool_calls": [{"id": "call_1", "type": "function",
"function": {"name": "search_tickets", "arguments": "{}"}}]},
{"role": "tool", "tool_call_id": "call_1",
"content": "3 open: NYX-204, NYX-219, NYX-231"}
]
}'