Reliability

Run Cruxy in production: per-tier deadlines, which errors to retry and how, Idempotency-Key, telling a finished stream from a failed one, and status.

This page collects what the gateway guarantees when a call is slow or fails, and what it leaves to your client. Each section links to the page that covers its topic in full. The sample streams a long kavi turn, accepts the answer only once the stream finishes, and retries a failed attempt:

# One key per logical request: reuse it if you send this request again.
IDEMPOTENCY_KEY=$(uuidgen)
 
# A long kavi turn: stream it. --max-time sits above the five-minute limit
# on a stream, so the gateway, not curl, decides when the turn has run long.
curl -sN --max-time 330 https://api.cruxy.ai/api/v1/chat \
  -H "Authorization: Bearer $CRUXY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -d '{
    "model": "kavi",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Plan a three-day walking tour of Old Delhi." }
    ]
  }' > stream.txt
 
# The completion finished only if the stream ended with [DONE].
grep -q '^data: \[DONE\]' stream.txt && echo "complete" || echo "incomplete: send it again"

// Your retry budget and timeout. Tune them for your workload; they are not API
// limits. A stream may run for up to five minutes, so the timeout sits above it.
const MAX_ATTEMPTS = 3
const BASE_DELAY_MS = 500
const TIMEOUT_MS = 330_000
 
// Transient failures the API marks retryable.
const RETRYABLE = new Set([
  'rate_limited',
  'credits_unavailable',
  'headless_concurrency_limit',
  'limits_unavailable',
  'upstream_overloaded',
  'model_unavailable',
  'all_models_unavailable',
  'server_shutting_down',
  'internal_error',
])
 
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
// Exponential backoff with full jitter.
const backoff = (attempt: number) => Math.random() * BASE_DELAY_MS * 2 ** attempt
 
type Attempt = { answer: string } | { retryInMs: number }
 
async function streamOnce(
  body: Record<string, unknown>,
  idempotencyKey: string,
  attempt: number,
): Promise<Attempt> {
  let res: Response
  try {
    res = await fetch('https://api.cruxy.ai/api/v1/chat', {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
        'Content-Type': 'application/json',
        'Idempotency-Key': idempotencyKey,
      },
      body: JSON.stringify({ ...body, stream: true }),
      signal: AbortSignal.timeout(TIMEOUT_MS),
    })
  } catch {
    return { retryInMs: backoff(attempt) } // no response at all: send it again
  }
 
  if (!res.ok) {
    const { code, error, request_id } = await res.json()
    if (!RETRYABLE.has(code)) throw new Error(`${res.status} ${code}: ${error} (request_id ${request_id})`)
    const retryAfter = Number(res.headers.get('Retry-After'))
    return { retryInMs: retryAfter > 0 ? retryAfter * 1000 : backoff(attempt) }
  }
  if (!res.body) return { retryInMs: backoff(attempt) }
 
  const reader = res.body.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  let answer = ''
  try {
    while (true) {
      const { value, done } = await reader.read()
      if (done) break
      buffer += decoder.decode(value, { stream: true })
      const lines = buffer.split('\n')
      buffer = lines.pop() ?? '' // keep a partial line for the next read
 
      for (const raw of lines) {
        const line = raw.trim()
        if (!line.startsWith('data:')) continue
        const data = line.slice('data:'.length).trim()
        if (data === '[DONE]') return { answer } // the only success signal
 
        const choice = JSON.parse(data).choices[0] // undefined on the usage frame
        if (choice?.delta.content) answer += choice.delta.content
      }
    }
  } catch {
    // The connection dropped mid-stream: fall through and treat it as a failure.
  }
 
  // The stream ended before [DONE]: a failed completion. Discard the partial answer.
  return { retryInMs: backoff(attempt) }
}
 
async function chat(body: Record<string, unknown>): Promise<string> {
  // One key per logical request, reused on every retry of it: if two attempts
  // both complete, the request is billed once.
  const idempotencyKey = crypto.randomUUID()
 
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    const result = await streamOnce(body, idempotencyKey, attempt)
    if ('answer' in result) return result.answer
    if (attempt < MAX_ATTEMPTS) await sleep(result.retryInMs)
  }
  throw new Error(`no complete answer after ${MAX_ATTEMPTS} attempts`)
}
 
const answer = await chat({
  model: 'kavi',
  messages: [{ role: 'user', content: 'Plan a three-day walking tour of Old Delhi.' }],
})
console.log(answer)

Deadlines and timeouts

With stream: false, the gateway makes one attempt at the model, bounded by the tier's deadline:

An attempt that runs past its deadline is cancelled and answered 504 upstream_timeout. Nothing is billed, and the gateway does not retry it, since a turn that needed longer needs longer again. The same deadlines apply on all three chat wires: POST /api/v1/chat, POST /api/v2/chat, and POST /v1/chat/completions.

• Stream long turns. A stream may run for up to five minutes. A long kavi turn, with reasoning on and a large output, belongs on a stream.
• Or ask for less. A lower max_tokens or reasoning_effort shortens the turn; see Reasoning effort.
• Set your client timeout above the deadline, or above five minutes for a stream, so a slow model reaches you as a 504 with a request_id rather than as your own abort.

What the gateway retries for you

• A failing model, inside the gateway. An overloaded or failing model, or a dropped connection to it, is retried a bounded number of times, with backoff, before the gateway answers 503 upstream_overloaded or model_unavailable. On a stream, that happens only while the stream opens, never after a byte has reached you.
• Never a non-streaming timeout. See Deadlines and timeouts.
• Never on another tier. Once a call is sent, it stays on its tier. model: "auto" picks a healthy tier before the call, and an explicit tier that keeps failing is refused at once with 503 model_unavailable while it cools down. See Auto routing.

What to retry

Branch on the error's code, never on its message:

When you retry:

• Honour Retry-After. Every 429 sends it, and so does a 503 limits_unavailable.
• Otherwise back off exponentially, with jitter, so many clients that failed together do not retry together.
• Cap the attempts and the total wait, so a persistent failure ends instead of hanging.
• Log the request_id of every failure, retried or not; see Request IDs and metadata.

Retry safely with Idempotency-Key

Send an Idempotency-Key header, one per logical request, and reuse it on every retry of that request:

• A success is stored for 24 hours. A later or concurrent call with the same key and the same body replays it byte for byte, without calling the model and without billing again. A concurrent duplicate waits for the first call to finish.
• Only a success is stored. An error, a 429 or any 5xx included, releases the key, so a retry with the same key and body runs the request again. That is the intended way to retry.
• A different body under the same key is 409 idempotency_conflict.
• A response over 256 KiB is served but not stored, so a retry runs again. It is still billed once.
• A key is at most 255 characters, and is scoped to your API key and the endpoint.
• A stream stores and replays nothing. A retry with the same key and body streams a new generation. When both attempts complete, the request is billed once; an attempt that did not complete is billed for the tokens it delivered, and the retry is billed as well.

Errors and retries lists the endpoints that accept the header.

Confirm a stream finished

Once a stream opens, its 200 is already sent, so a failure after that point cannot change the status. How the stream tells you it failed depends on the wire:

• POST /api/v1/chat: data: [DONE] is the only success signal. A failure after the stream opens sends no error frame; the stream stops early. Treat a stream that ends before [DONE] as a failed completion: discard the partial output or retry, and never show it as a finished answer.
• POST /api/v2/chat: the stream always ends with exactly one finish or error frame. It has no [DONE].
• POST /v1/chat/completions: a failure after the stream opens ends it with a data: {"error": {...}} frame instead of [DONE].

The tokens a failed stream delivered are billed as a partial usage row. Streaming shows how to detect each case.

Check platform status

status.getcruxy.com is the Cruxy status page. For the same picture in code, GET /api/status returns a status document. It takes no key.

curl https://api.cruxy.ai/api/status

const res = await fetch('https://api.cruxy.ai/api/status')
 
// A 503 still carries the document, with status "unknown": no model component
// has a fresh measurement, so the document cannot be trusted.
const status = await res.json()
console.log(`platform: ${status.status}`)
 
// Iterate the array and read each row's kind; new components are appended.
for (const component of status.components) {
  console.log(`${component.id} (${component.kind}): ${component.status}`)
}

• status is the whole platform in one word: healthy, degraded, down, or unknown.
• components has a row for the API, the database, each chat tier, and the cruxy-embed, cruxy-rerank, and cruxy-guard models, each with its own status and when it was last checked. A component that is measurably down is a 200 carrying down; the document is a 503 only when no model component has a fresh measurement.
• The verdicts come from active probes, not from live traffic, and the response is never cached.

GET /api/health reports only that the gateway's HTTP listener is up. It checks no model or database, so do not read it as platform status.

Production checklist

• Use a developer API key on a server. See Get your API key.
• Pin Cruxy-Version, and ignore JSON fields you do not recognize.
• Set client timeouts above the tier deadline, or above five minutes for a stream.
• Stream long kavi turns, and size max_tokens for the thinking as well as the answer.
• Send an Idempotency-Key on every request that takes one, and reuse it on that request's retries.
• Retry only what the table above says, honouring Retry-After, with a cap on attempts.
• Check for [DONE], or a finish frame on v2, before you show a streamed answer.
• Log the X-Request-Id, the request_id, and the Cruxy-Version of every call.
• Read Known limitations before you launch.

Next steps

• Errors and retries: every error code, and a retrying client.
• Streaming: the frames, and a complete stream reader.
• Rate limits and budgets: read your headroom before you hit a 429.
• Known limitations: what the API does not do yet, and the workarounds.