Streaming

Read chat completions as Server-Sent Events: the frame order, the chunk and usage shapes, and how to tell a finished stream from a failed one.

stream is optional and false by default. With stream: true, POST /api/v1/chat answers with text/event-stream: OpenAI-style Server-Sent Events in which every event is one chat.completion.chunk JSON object on a data: line. The stream ends with the literal line data: [DONE], and that line is the only signal that the completion finished.

curl -N https://api.cruxy.ai/api/v1/chat \
  -H "Authorization: Bearer $CRUXY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vaani",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Write a haiku about monsoon rain." }
    ]
  }' | tee stream.txt
 
# The completion finished only if the stream ended with [DONE].
grep -q '^data: \[DONE\]' stream.txt || echo "stream failed before [DONE]"

const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'vaani',
    stream: true,
    messages: [{ role: 'user', content: 'Write a haiku about monsoon rain.' }],
  }),
})
if (!res.ok || !res.body) throw new Error(`chat failed: ${res.status} ${await res.text()}`)
 
const reader = res.body.getReader()
const decoder = new TextDecoder()
let buffer = ''
let text = ''
let finishReason: string | null = null
let usage: Record<string, unknown> | undefined
let sawDone = false
 
read: while (true) {
  const { value, done } = await reader.read()
  if (done) break
  buffer += decoder.decode(value, { stream: true })
 
  const lines = buffer.split('\n')
  buffer = lines.pop() ?? '' // keep a partial line for the next read
 
  for (const raw of lines) {
    const line = raw.trim()
    if (!line.startsWith('data:')) continue
    const data = line.slice('data:'.length).trim()
 
    if (data === '[DONE]') {
      sawDone = true
      break read
    }
 
    const chunk = JSON.parse(data)
    if (chunk.usage) usage = chunk.usage
 
    const choice = chunk.choices[0] // undefined on the usage frame
    if (!choice) continue
    if (choice.delta.content) text += choice.delta.content
    if (choice.finish_reason) finishReason = choice.finish_reason
  }
}
 
if (!sawDone) {
  // The stream ended early: a failed completion, not a short one.
  throw new Error('stream ended before [DONE]; discard the partial output')
}
 
console.log(text)
console.log(finishReason, usage)

Frame order

A successful stream sends these frames, in this order:

On the wire:

data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}],"cruxy":{"routed_tier":"vaani","request_id":"a1b2c3d4-e5f6-7890-abcd-ef1234567890","reasoning_effort":"low"}}
 
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"content":"Clouds lean"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"content":" on the hills"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
 
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":48,"total_tokens":60,"cost":{"amount":<amount>,"currency":"<currency>","tier":"vaani",...}}}
 
data: [DONE]

Changing the framing or this order is a breaking change that ships only under a new dated Cruxy-Version, so you can rely on it. New cruxy.* fields can appear without a version change, so ignore fields you do not recognize.

The chunk

The role primer's content is an empty string, not absent. Appending every content you see is safe; the primer adds nothing.

Tool-call deltas

When the model calls a tool, deltas carry tool_calls fragments instead of content. Every fragment has an index and a piece of function.arguments, a JSON string; fragments can also carry the call's id, type, and function.name. Assemble the fragments per index, concatenating the arguments pieces. A finish frame with finish_reason: "tool_calls" means the fragments are complete: run the calls and answer them as Handle tool calls describes.

The usage frame

The usage frame is the only frame with usage, and it is never merged into the finish frame before it, because an OpenAI-compatible client reads usage only from the frame whose choices is empty.

A stream names its token fields the OpenAI way (prompt_tokens, completion_tokens); a non-streaming response uses input_tokens and output_tokens. Usage and cost covers both shapes, the cost object, and which tokens are billed.

[DONE] is the only success signal

Once the stream opens, the 200 is already sent. If the completion fails after that, no error frame follows: the gateway stops sending and closes the response without the finish frame, the usage frame, or data: [DONE].

To detect it on the client:

1. Set a flag when you read data: [DONE], as the sample above does with sawDone.
2. When the read loop ends, treat a missing flag as a failure.
3. Treat an exception from the read itself, such as a dropped connection, as a failure too.

A stream that ends with [DONE] is complete, but the answer can still be cut short: finish_reason: "length" means the output reached max_tokens or the tier's cap. On kavi, which reasons by default, a small max_tokens can be spent entirely on thinking; see Features overview.

The gateway retries a failing model only while the stream opens, never after a byte has reached you. A stream may run for up to five minutes.

A stream stores and replays nothing under Idempotency-Key, since a stream cannot be replayed: a retry with the same key and body streams a new generation. When both attempts complete, the request is billed once. An attempt that did not complete is billed for the tokens it delivered, and the retry is billed as well.

Other chat wires

The rules above are for POST /api/v1/chat. The other two chat endpoints end their streams differently:

• POST /api/v2/chat sends named events (start, reasoning, delta) and always ends with exactly one finish or error frame. It has no [DONE]; a stream that closes without finish or error failed.
• POST /v1/chat/completions ends a stream that fails after it opens with a data: {"error": {...}} frame instead of [DONE]. It sends the usage frame only when the request sets stream_options: {"include_usage": true}.

To render a stream in a browser without exposing your key, see Streaming chat in the browser.

Next steps

• Chat: the request options and the non-streaming response.
• Reliability: retry a stream that failed, and the deadlines it avoids.
• Handle tool calls: answer the tool calls a stream returns.
• Errors and retries: what to retry, and how.