CruxyLearn

Streaming chat in the browser

Consume the OpenAI-style SSE stream in a browser, render deltas without jank, and recover when the connection breaks mid-response.

A streaming chat UI reads tokens as the model writes them. This recipe proxies Chat with stream: true through your own server (so the key never reaches the browser), then consumes the Server-Sent Events on the client, painting each delta as it arrives.

1. Proxy the stream past your key

A browser must not hold CRUXY_API_KEY. Put a thin server route in front: it adds the bearer key, requests the stream from the gateway, and pipes the SSE body straight through to the client unchanged.

// server route, e.g. POST /api/chat
export async function POST(req: Request) {
  const { prompt } = await req.json()
 
  const upstream = await fetch('https://api.cruxy.ai/api/v1/chat', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'kavi',
      stream: true,
      messages: [{ role: 'user', content: prompt }],
    }),
  })
 
  // Forward the raw event stream; the browser reads the same chunks the
  // gateway emitted, terminated by `data: [DONE]`.
  return new Response(upstream.body, {
    headers: { 'Content-Type': 'text/event-stream' },
  })
}

2. Consume the SSE in the browser

Read the response body as a stream, split it on the blank line between events, and pull each token from choices[0].delta.content. Tokens land in a buffer that a single requestAnimationFrame flushes to the DOM, so a fast stream repaints once per frame instead of once per token.

async function streamInto(el: HTMLElement, prompt: string) {
  const res = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
  })
  if (!res.ok || !res.body) throw new Error(`chat failed: ${res.status}`)
 
  const reader = res.body.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  let pending = ''
  let frame = 0
  let done = false
 
  // Batch DOM writes: deltas accumulate in `pending`, one rAF flush paints.
  const flush = () => {
    if (pending) {
      el.textContent += pending
      pending = ''
    }
    frame = 0
  }
  const schedule = () => {
    if (!frame) frame = requestAnimationFrame(flush)
  }
 
  try {
    while (true) {
      const { value, done: streamDone } = await reader.read()
      if (streamDone) break
      buffer += decoder.decode(value, { stream: true })
 
      const events = buffer.split('\n\n')
      buffer = events.pop() ?? '' // keep the trailing partial event
 
      for (const event of events) {
        const line = event.replace(/^data: /, '').trim()
        if (!line) continue
        if (line === '[DONE]') {
          done = true
          continue
        }
        const chunk = JSON.parse(line)
        const delta = chunk.choices?.[0]?.delta?.content
        if (delta) {
          pending += delta
          schedule()
        }
      }
    }
  } catch {
    // The stream broke after the 200 arrived; see "handling mid-stream
    // failure" below. Surface what streamed so far, then let the caller retry.
    el.dataset.error = 'stream interrupted'
  } finally {
    if (frame) cancelAnimationFrame(frame)
    flush() // paint whatever landed before the stream ended
  }
 
  // A clean HTTP close is not the same as a complete answer: only `[DONE]`
  // marks completion. No sentinel means the response was cut short.
  if (!done) el.dataset.incomplete = 'true'
}

The streaming shape differs from non-streaming

A streaming chunk is not shaped like the non-streaming reply. With stream: true, each chat.completion.chunk carries an OpenAI choices[] array and the token lives at choices[0].delta.content. With stream: false, the gateway flattens the reply to a top-level message and there is no choices[] at all. Code written against the flattened message.content will not find a token in a stream chunk, and vice versa; Chat documents both shapes side by side.

The first chunk is a role primer whose delta.content is an empty string; the if (delta) guard skips it. The final chunk before [DONE] carries a usage object and an empty choices array, which the optional chaining on choices?.[0] skips; read usage there if you want token counts and cost for the call. Streaming documents every frame in order.

Rendering without jank

The reader hands you tokens faster than a browser needs to repaint. Writing to the DOM on every delta forces a layout on each one and the text visibly stutters. Accumulating deltas in a buffer and flushing once per requestAnimationFrame collapses a burst of tokens into a single repaint, so the text advances smoothly however fast the stream runs.

Handling mid-stream failure

The response status resolves before any token streams. Once the gateway has sent 200, a later failure (a dropped connection, an upstream cut) cannot change that status; it surfaces as reader.read() throwing or as the stream ending. Two signals cover it: the try/catch catches a thrown read, and the [DONE] sentinel confirms completion. A stream that ends without [DONE] delivered a partial answer even though nothing threw, so this recipe marks it incomplete. Treat either signal as a cue to keep the partial text and offer a retry.

Next steps

Was this page helpful?