# Cruxy Learn: full corpus
> OpenAI-compatible LLM gateway for the Cruxy model tiers (kavi, vaani, and mira), plus embeddings, rerank, and content moderation behind one base URL (https://api.cruxy.ai) and one Bearer key.
Every documentation page, concatenated in navigation order. Each block below is fenced by a rule stating the page title, its canonical URL, and the markdown source it was rendered from. The per-page link index is at https://learn.getcruxy.com/llms.txt.
================================================================================
Page: Start here
URL: https://learn.getcruxy.com/learn/en/start-here
Markdown: https://learn.getcruxy.com/learn/en/start-here.md
================================================================================
# Start here
A reading path through the Cruxy API in nine pages, from your first call to tool calling, retrieval, moderation, and running in production.
Each step names one page and what you take from it. The pages build on each
other, so read them in order the first time; after that, each one stands on
its own as a reference.
1. **Make your first call**
[Quickstart](/docs/en/get-started/quickstart) takes you from an empty
terminal to your first model response: you get an API key with device login,
then send a chat request to the `kavi` tier.
2. **Choose a tier**
[Choosing a tier](/docs/en/build/choosing-a-tier) compares the tiers you pass
as `model` (`kavi`, `vaani`, and `mira`) by profile, output cap, and how each
one reasons by default, and shows how to read the live catalog from
`GET /api/v1/models` instead of hardcoding it.
3. **Hold a conversation**
[Chat](/docs/en/build/chat) documents `POST /api/v1/chat`, the
OpenAI-compatible chat completion: the request options, a single JSON
completion or a Server-Sent Events stream, the `cruxy` block, and usage.
4. **Let the model call your functions**
[Tutorial: build a tool-using agent](/docs/en/build/tool-agent) builds on the
same endpoint. You describe two functions, run the calls the model returns,
send the results back, and loop on `finish_reason` until the model answers.
5. **Answer from your own documents**
[Retrieval-augmented generation](/docs/en/resources/cookbook/rag) is a
cookbook recipe that chains three endpoints: embeddings for a first-pass
search, rerank to sharpen the shortlist, and chat to write the answer.
6. **Screen content**
[Moderations](/docs/en/build/moderations) checks text against the
content-safety categories with the `cruxy-guard` model and returns an
overall `flagged` result.
7. **Handle failures**
[Errors and retries](/docs/en/build/errors-and-retries) defines the error
envelope, lists every error code, and says which failures to retry and how
long to wait, so your client branches on the stable `code`, never on the
wording of the message.
8. **Run it reliably**
[Reliability](/docs/en/build/reliability) puts the production rules in one
place: the per-tier deadlines, which failures to retry, `Idempotency-Key`, how
to tell a finished stream from a failed one, and where to check platform
status.
9. **Know the limits**
[Known limitations](/docs/en/build/known-limitations) lists what the API does
not do today, such as returning `kavi`'s thinking or running a non-streaming
call past 90 seconds, and the workaround for each.
## Next steps
[Cookbook](/docs/en/resources/cookbook) has more recipes that combine
endpoints to finish a real job, and [API Reference](/docs/en/reference/api)
lists every endpoint with its exact request and response shape.
Moving code that already calls OpenAI?
[Migrate from OpenAI](/docs/en/migrate/from-openai) changes the base URL, the
key, and `model`, and
[OpenAI SDK compatibility](/docs/en/migrate/openai-sdk-compatibility) says what
carries over.
================================================================================
Page: One gateway for every model tier
URL: https://learn.getcruxy.com/docs/en
Markdown: https://learn.getcruxy.com/docs/en.md
================================================================================
# One gateway for every model tier
An OpenAI-compatible LLM gateway for the Cruxy model tiers: chat, embeddings, and moderation behind one API key.
## Choose how you build
Two ways into the same gateway: call the endpoints directly, or start from a
pattern and copy what fits.
- [Chat](/docs/en/build/chat)
- [Embeddings](/docs/en/build/embeddings)
- [Rerank](/docs/en/build/rerank)
- [Moderations](/docs/en/build/moderations)
- [Feedback](/docs/en/build/feedback)
- [API Reference](/docs/en/reference/api)
- [Cookbook](/docs/en/resources/cookbook)
- [API guide for agents](/docs/en/resources/api-guide)
- [Glossary](/docs/en/resources/glossary)
- [Changelog](/docs/en/reference/changelog)
## From key to production
Every integration walks the same path; each stage links into the guide that
covers it. To read it as one ordered path, follow
[Start here](/learn/en/start-here).
- [Features overview](/docs/en/get-started/features)
- [Quickstart](/docs/en/get-started/quickstart)
- [Get your API key](/docs/en/get-started/api-keys)
- [Authentication](/docs/en/get-started/authentication)
- [Chat](/docs/en/build/chat)
- [Streaming](/docs/en/build/streaming)
- [Define tools](/docs/en/build/tools)
- [Embeddings](/docs/en/build/embeddings)
- [Rerank](/docs/en/build/rerank)
- [Moderations](/docs/en/build/moderations)
- [Models](/docs/en/reference/models)
- [Errors and retries](/docs/en/build/errors-and-retries)
- [Rate limits and budgets](/docs/en/build/rate-limits)
- [Feedback](/docs/en/build/feedback)
- [API Reference](/docs/en/reference/api)
## Pick a model
You pass a tier id as `model` (or `auto` to let the router choose), and the
gateway routes the call. The tier is the contract; the upstream model behind it
is never exposed. [Choosing a tier](/docs/en/build/choosing-a-tier) compares
them, and [Models](/docs/en/reference/models) documents every id and alias.
- `kavi` (Deep): Deepest reasoning and the largest output budget; reach for it on the hardest tasks.
- `vaani` (Balanced): Strong quality at lower latency and cost, a sensible default for everyday calls.
- `mira` (Fast): Lowest latency and cost for short, high-volume calls; the budget-downgrade target.
## Start from a recipe
Four working patterns that combine the endpoints to do a real job; copy the
one shaped like yours. [Cookbook](/docs/en/resources/cookbook) indexes them
all.
## Start building
One key, one endpoint, three tiers behind it.
================================================================================
Page: Features overview
URL: https://learn.getcruxy.com/docs/en/get-started/features
Markdown: https://learn.getcruxy.com/docs/en/get-started/features.md
================================================================================
# Features overview
What the Cruxy gateway gives you: OpenAI-compatible chat on three tiers, specialized model endpoints, and memory and rules you control.
Cruxy is an OpenAI-compatible LLM gateway. You call one API at
`https://api.cruxy.ai` with one key and name a model tier instead of a provider
model. Cruxy-native fields are additive and namespaced under a `cruxy` key, so
an OpenAI client keeps parsing the response and ignores what it does not know.
## Three tiers and auto
You pass a tier id as the chat `model`. The gateway resolves the tier to an
upstream model on the server; an upstream provider model is never accepted or
returned, so the tier is the stable contract your code depends on.
| Tier | Profile | Default reasoning | Reasoning control |
| --- | --- | --- | --- |
| `kavi` | Deep | `high` | On or off: any value except `none` turns thinking on. |
| `vaani` | Balanced | `low` | Graded: `low`, `medium`, and `high` are different depths. |
| `mira` | Fast | `none` | Graded: `low`, `medium`, and `high` are different depths. |
The reasoning columns are the shipped defaults. `GET /api/v1/models` reports
each tier's live `reasoning.default_effort` and `reasoning.graded`, along with
its context window, output cap, and pricing; [Models](/docs/en/reference/models)
shows how to read the catalog at runtime, and
[Choosing a tier](/docs/en/build/choosing-a-tier) compares the tiers.
A reasoning tier thinks inside the same `max_tokens` budget as its answer, and
the thinking is not returned as content. A small `max_tokens` on `kavi` can be
spent entirely on thinking, ending with `finish_reason: "length"` and empty
content. Send `reasoning_effort: "none"` for a short, direct answer, or leave
`max_tokens` unset to keep the tier's cap; see
[Reasoning effort](/docs/en/build/reasoning-effort).
Send `model: "auto"` to let the server-side router pick the tier from the
request. It resolves to a concrete tier before billing and before the upstream
call, and the tier that served the request comes back as `cruxy.routed_tier`.
An `auto` request may be budget-downgraded to `mira`, so read `routed_tier`
rather than assuming it; [Auto routing](/docs/en/build/auto-routing) covers the
router, and [Rate limits and budgets](/docs/en/build/rate-limits) explains the
budgets behind the downgrade.
## What you can call
| Area | Endpoints | What it does |
| --- | --- | --- |
| [Chat](/docs/en/build/chat) | `POST /api/v1/chat` | OpenAI-compatible chat with Cruxy features: [Streaming](/docs/en/build/streaming), tool calling (see [Define tools](/docs/en/build/tools)), [Memory](/docs/en/build/memory), [Standing rules](/docs/en/build/rules), and attached files. |
| Native chat | `POST /api/v2/chat` | The Cruxy-native wire: named SSE events that always end in a `finish` or `error` frame. |
| OpenAI wire | `POST /v1/chat/completions`, `GET /v1/models` | The plain OpenAI protocol for OpenAI clients and evaluation harnesses, with none of the Cruxy-native features. |
| [Models](/docs/en/reference/models) | `GET /api/v1/models` | The tier catalog. |
| Token counting | `POST /api/v1/tokens/count` | The input tokens a chat request would bill, counted without inference and not billed. |
| [Embeddings](/docs/en/build/embeddings) | `POST /api/v1/embeddings` | 1024-dimension vectors from `cruxy-embed`. |
| [Rerank](/docs/en/build/rerank) | `POST /api/v1/rerank` | Documents ranked against a query by `cruxy-rerank`. |
| [Moderations](/docs/en/build/moderations) | `POST /api/v1/moderations` | Content classification by `cruxy-guard`. |
| [Feedback](/docs/en/build/feedback) | `POST /api/v1/feedback` | A signal about an earlier call, joined to it by `request_id`. |
| Files | `/api/v1/files` | Uploads that a chat message attaches with `file_ids`; chat reads text, PDF, and Word (`.docx`) files. |
| Conversations | `/api/v1/conversations` | Conversation history stored server-side, private to its author, with branches. |
| Memory | `/api/v1/memories`, `/api/v1/vaults` | Stored facts, personal, organization-wide, or shared in a vault, that chat can recall. |
| Rules | `/api/v1/rules`, `/api/v1/rules/resolved` | Standing instructions for chat, set for your organization or a vault. |
| Prompts | `/api/v1/prompts` | Named prompts with immutable versions. |
| Evals | `/api/v1/evals/suites` | Test suites run against a tier; every case is a billed completion. |
| Account | `/api/v1/me`, `/api/v1/profile`, `/api/v1/limits` | Your identity, profile, and live limits. |
| Organization | `/api/v1/org/members`, `/api/v1/workspaces`, `/api/v1/logs` | Members, invitations, workspaces with spend caps, and request-log metadata. |
| Sign-in | `/api/v1/device/code`, `/api/v1/device/token` | Device login, which issues API keys; see [Get your API key](/docs/en/get-started/api-keys). |
| Status | `GET /api/status`, `GET /api/health`, `GET /api/ready` | Public platform status and health probes. |
Every `/api/v1` endpoint is also served under `/api/v2` with identical
behavior, and this site documents the `/api/v1` path. Chat is the one
exception: `POST /api/v2/chat` is a different wire from `POST /api/v1/chat`.
[API Reference](/docs/en/reference/api) has every operation with its exact
request and response shape.
## Memory and standing rules
Two chat features add context beyond the messages you send. You control both
per request, and their defaults depend on how the key was issued:
- **On by default** for a signed-in credential: what a person gets by signing
in to the Cruxy web app, desktop, the CLI, or mobile. A key minted by device
login or OAuth login is a signed-in credential whichever client ran the flow,
a third-party tool included.
- **Off by default** for a developer API key or a headless key, which opts in
per request.
`GET /api/v1/limits` reports which kind a key is as `bucket`: `subscription` is
a signed-in credential, and `apikey` and `headless` are not. A server, or
anything that serves people other than the key's owner, should use a developer
API key; see [Get your API key](/docs/en/get-started/api-keys#two-kinds-of-key).
To get the same behavior from any key, set both on every request.
### Memory
Memory has two independent sides, each set in the request's `memory` object:
- **Recall** (`recall`) adds the stored memories most relevant to the turn as
background context and reports them in `cruxy.memory_recall`: the scope and
key of each, never the stored value.
- **Capture** (`capture`) lets the gateway store a durable fact from the turn,
such as a stated preference, and reports it in `cruxy.memory_capture`. No
field means nothing was stored. Captures go to your personal memory, or to a
vault you can write when the request names one, never to organization-wide
memory. Content carrying an email address, phone number, or card number is
dropped before any write.
`true` needs a key whose allowed scopes include memory; otherwise the request
fails with `403` `key_scope_forbidden`. `false` needs no scope. Pin
`Cruxy-Version: 2026-09-26` on every request that relies on `false`, so a
gateway that predates the field refuses the request instead of ignoring it.
`GET /api/v1/memories` lists what is stored, and
`DELETE /api/v1/memories/{id}` removes an entry. [Memory](/docs/en/build/memory)
covers recall, capture, and the scope a key needs.
### Standing rules
Standing rules are instructions your organization sets for chat, and a vault
can have its own. They are injected into the turn as defaults that the user's
own message overrides. A developer API key gets them only when the request sends
`rules: true`, or when the key's allowed scopes name the rules family. Any key
may send `rules: false` to decline them for a turn.
When rules apply, the response carries `cruxy.rules_applied` with counts and
sources, never the rule text. `GET /api/v1/rules/resolved` returns the exact
rules a turn would get. Injected rule text counts as input tokens.
[Standing rules](/docs/en/build/rules) covers opting in and what gets
injected.
This request recalls memory, stores nothing new, and declines standing rules:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-H "Cruxy-Version: 2026-09-26" \
-d '{
"model": "vaani",
"stream": false,
"memory": { "recall": true, "capture": false },
"rules": false,
"messages": [
{ "role": "user", "content": "Which time zone should my meeting use?" }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
'Cruxy-Version': '2026-09-26',
},
body: JSON.stringify({
model: 'vaani',
stream: false,
memory: { recall: true, capture: false },
rules: false,
messages: [{ role: 'user', content: 'Which time zone should my meeting use?' }],
}),
})
const completion = await res.json()
console.log(completion.message.content)
console.log(completion.cruxy.memory_recall) // undefined when nothing was recalled
```
## Versioning
The API is versioned by date. Send `Cruxy-Version: YYYY-MM-DD` to pin a
version, or omit it for the server's current default; an unknown or malformed
value is rejected with `400`. Every response, errors included, echoes the
resolved version in its `Cruxy-Version` header.
A breaking change, such as a change to streaming frame order, ships under a new
dated version. Additive fields, including new `cruxy.*` fields, ship without
one, so ignore JSON fields you do not recognize.
[Changelog](/docs/en/reference/changelog) records what changed.
## Next steps
- [Get your API key](/docs/en/get-started/api-keys): mint the key every call needs.
- [Quickstart](/docs/en/get-started/quickstart): make your first chat call.
- [Streaming](/docs/en/build/streaming): read a chat completion as it is written.
- [Errors and retries](/docs/en/build/errors-and-retries): handle failures.
================================================================================
Page: Quickstart
URL: https://learn.getcruxy.com/docs/en/get-started/quickstart
Markdown: https://learn.getcruxy.com/docs/en/get-started/quickstart.md
================================================================================
# Quickstart
Get an API key with device login, then make your first chat call.
You'll go from an empty terminal to your first model response against the
`kavi` tier. Every request in this guide goes to `https://api.cruxy.ai`.
## 1. Start device login
Ask the gateway for a device code. This endpoint needs no authentication.
```bash
curl -X POST https://api.cruxy.ai/api/v1/device/code
```
You get back a `user_code` to type, a URL to approve it, and the
`device_code` your client polls with.
```json
{
"device_code": "cxy_dev_QK3f...redacted",
"user_code": "WDJB-MJHT",
"verification_uri": "https://cruxy.ai/device",
"verification_uri_complete": "https://cruxy.ai/device?user_code=WDJB-MJHT",
"expires_in": 600,
"interval": 5
}
```
## 2. Approve in the browser
Open `verification_uri_complete` and approve the request. The
`user_code` (`WDJB-MJHT`) is pre-filled; if you open the plain
`verification_uri` instead, type it in by hand. You have `expires_in`
seconds (600 = ten minutes) to finish.
## 3. Poll for the key
Poll the token endpoint with your `device_code`, waiting `interval`
seconds (5) between attempts.
```bash
curl -X POST https://api.cruxy.ai/api/v1/device/token \
-H "Content-Type: application/json" \
-d '{ "device_code": "cxy_dev_QK3f...redacted" }'
```
> **Note:** Until someone approves, you get `400` with an `authorization_pending`
> code. That is expected; keep polling.
```json
{
"code": "authorization_pending",
"error": "authorization_pending",
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
```
Once approved, the same request returns `200` with your key.
> **Warning:** The key is shown exactly once; there is no way to retrieve it again.
> Copy it now and store it before you move on.
```json
{
"access_token": "cxy_live_...redacted",
"token_type": "bearer",
"key_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"user_id": "6f9c1e2a-1b2c-4d5e-8f90-1a2b3c4d5e6f",
"expires_at": "2026-10-20T12:00:00.000Z",
"org": {
"id": "8c1a5b3e-2d4f-4a6b-9c8d-0e1f2a3b4c5d",
"name": "Acme Corp",
"role": "member"
}
}
```
Export it into your shell. Never commit a key to source control.
```bash
export CRUXY_API_KEY="cxy_live_..."
```
For the full flow (every polling error code, verifying the key, and key
hygiene), see [Get your API key](/docs/en/get-started/api-keys).
## 4. Make your first call
Send a prompt to the `kavi` tier. `stream: false`, which is also the default,
asks for a single JSON response instead of a stream, so there is one block to
read on a first call.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'kavi',
stream: false,
messages: [{ role: 'user', content: 'Hello' }],
}),
})
const completion = await res.json()
console.log(completion)
```
The response is an OpenAI-shaped chat completion. Alongside the standard
fields, Cruxy adds a namespaced `cruxy` block carrying the `request_id`
and the cost of the call; a vanilla OpenAI client ignores it. See
[Chat](/docs/en/build/chat) for the streaming form and the full request options.
## Where to go next
- [Chat](/docs/en/build/chat): Streaming SSE, tiers, temperature, and the cruxy block.
- [Get your API key](/docs/en/get-started/api-keys): The device-login flow end to end, the key format, and key hygiene.
- [API Reference](/docs/en/reference/api): Every endpoint, typed and interactive.
================================================================================
Page: Get your API key
URL: https://learn.getcruxy.com/docs/en/get-started/api-keys
Markdown: https://learn.getcruxy.com/docs/en/get-started/api-keys.md
================================================================================
# Get your API key
Pick the right kind of key, mint a signed-in key with device login, verify it with GET /api/v1/me, and know when it expires.
Every authenticated request sends a `cxy_live_` user key, and a user key is one
of two kinds. A signed-in credential comes from device login and belongs in a
person's own tool, such as a CLI or a desktop app. A developer API key is
created in the Cruxy Console and belongs on a server.
> **Warning:** Device login gives you a signed-in credential, with memory and standing rules on
> by default. For a server, or anything that serves people other than you, use a
> developer API key from the Cruxy Console. A signed-in key expires 90 days after
> it is issued, and today you cannot revoke it yourself.
## Two kinds of key
| | Signed-in credential | Developer API key |
| --- | --- | --- |
| How you get it | Device login, or native OAuth login, whichever client runs the flow. | Created in the Cruxy Console. |
| Use it for | A person's own tool, such as a CLI or a desktop app. | A server, and anything that serves people other than the key's owner. |
| Memory and rules | On by default. | Off unless a request opts in. |
| Expires | At `expires_at`, 90 days after issue. Use does not renew it. | Never. |
| Rotate or revoke | Not by its holder. | In the Console. |
| Spend limits | A monthly spend cap. | The Console can limit it to chosen models and a monthly spend cap. |
| Limits bucket | `subscription` | `apikey` |
A signed-in credential on a server would recall its owner's memory into other
people's turns, store what they say in that memory, and apply the owner's
organization rules, unless every request sends
`memory: {"recall": false, "capture": false}` and `rules: false`. A headless
key, billed from prepaid credit, has memory and rules off by default, like a
developer API key. To see which kind a key is, read `bucket` on
`GET /api/v1/limits`; see [Rate limits and budgets](/docs/en/build/rate-limits#read-your-limits).
## Device login
Cruxy issues signed-in credentials through an [RFC 8628][rfc]
device-authorization flow: your client asks for a code, a person approves it in
a browser, and your client polls until the key is minted. The browser and the
client can be different machines.
[rfc]: https://datatracker.ietf.org/doc/html/rfc8628
The two device-login endpoints take no key and no `Authorization` header: their
job is to issue one. Both are rate-limited and return `429` with
`rate_limited` if you call them too often.
1. **Request a device code**
`POST /api/v1/device/code` with no body.
```bash title="curl"
curl -X POST https://api.cruxy.ai/api/v1/device/code
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/device/code', {
method: 'POST',
})
if (!res.ok) throw new Error(`device code request failed: ${res.status}`)
const device = await res.json()
console.log(`Open ${device.verification_uri_complete} to approve.`)
console.log(`Confirm this code on that page: ${device.user_code}`)
```
```json
{
"device_code": "cxy_dev_QK3f...redacted",
"user_code": "WDJB-MJHT",
"verification_uri": "https://cruxy.ai/device",
"verification_uri_complete": "https://cruxy.ai/device?user_code=WDJB-MJHT",
"expires_in": 600,
"interval": 5
}
```
| Field | Meaning |
| --- | --- |
| `device_code` | The code your client polls with. The person approving never needs it. |
| `user_code` | The short code the person confirms in the browser. |
| `verification_uri` | The approval page. The person types `user_code` there. |
| `verification_uri_complete` | The approval page with `user_code` already filled in. |
| `expires_in` | Seconds until `device_code` expires. |
| `interval` | Seconds to wait between polls. |
Read `verification_uri`, `expires_in`, and `interval` from each response
rather than hardcoding the example values.
2. **Approve in the browser**
Send the person to `verification_uri_complete`, or to `verification_uri` with
the `user_code` to type. They approve the request before
`expires_in` seconds pass. If they decline, polling returns `access_denied`.
3. **Poll for the key**
`POST /api/v1/device/token` with the `device_code`, waiting `interval` seconds
between attempts. Polling faster than the limiter allows returns `429` with
`rate_limited`; wait the `Retry-After` seconds, then resume at `interval`.
```bash title="curl"
# DEVICE_CODE holds the device_code from step 1.
curl -X POST https://api.cruxy.ai/api/v1/device/token \
-H "Content-Type: application/json" \
-d "{\"device_code\": \"$DEVICE_CODE\"}"
```
```ts title="TypeScript"
// `device` is the response from step 1.
const sleep = (seconds: number) => new Promise((r) => setTimeout(r, seconds * 1000))
async function pollForKey(device: { device_code: string; interval: number; expires_in: number }) {
const deadline = Date.now() + device.expires_in * 1000
while (Date.now() < deadline) {
await sleep(device.interval)
const res = await fetch('https://api.cruxy.ai/api/v1/device/token', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ device_code: device.device_code }),
})
if (res.ok) return res.json()
if (res.status === 429) {
// Polling faster than the limiter allows: honour Retry-After, then resume.
await sleep(Number(res.headers.get('Retry-After')) || device.interval)
continue
}
const { code, reason } = await res.json()
if (code === 'authorization_pending') continue
throw new Error(`device login stopped: ${code}${reason ? ` (${reason})` : ''}`)
}
throw new Error('device_code expired before approval; start a new device login')
}
const key = await pollForKey(device)
// Store key.access_token in your secrets manager now. Log the id, never the key.
console.log(`Minted key ${key.key_id} for ${key.org.name}, expires ${key.expires_at}`)
```
Until the flow resolves, the endpoint returns `400` with an OAuth-style body
carrying `code`, `error`, and `request_id`:
```json
{
"code": "authorization_pending",
"error": "authorization_pending",
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
```
Only `authorization_pending` means keep polling:
| `code` | Meaning | What to do |
| --- | --- | --- |
| `authorization_pending` | Not approved yet. | Wait `interval` seconds and poll again. |
| `access_denied` | The person declined. | Stop. |
| `expired_token` | The `device_code` expired. | Start a new device login. |
| `invalid_grant` | The `device_code` is not valid. | Start a new device login. |
A refusal can also carry `reason` and `error_description`. The one `reason`
today is `org_membership_revoked`: the login was approved for an organization
the account no longer belongs to. A new login succeeds, but it may bind to a
different organization, so tell the person instead of retrying silently.
Branch on `reason`; `error_description` is for people and is not meant to be
parsed.
On approval the endpoint returns `200` with the key:
```json
{
"access_token": "cxy_live_...redacted",
"token_type": "bearer",
"key_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"user_id": "6f9c1e2a-1b2c-4d5e-8f90-1a2b3c4d5e6f",
"expires_at": "2026-10-20T12:00:00.000Z",
"org": {
"id": "8c1a5b3e-2d4f-4a6b-9c8d-0e1f2a3b4c5d",
"name": "Acme Corp",
"role": "member"
}
}
```
> **Warning:** The key is returned exactly once. Store it before you do anything else.
## What the token response tells you
| Field | Meaning |
| --- | --- |
| `access_token` | Your API key. Send it as `Authorization: Bearer `. |
| `token_type` | `bearer`. |
| `key_id` | The key's id. Safe to log. |
| `user_id` | The account the key belongs to. |
| `expires_at` | When the key stops working, 90 days after issue. Record it with the key; see [Expiry and revocation](#expiry-and-revocation). |
| `org` | The one organization this key is bound to: its `id`, `name`, and your `role` in it (`owner`, `admin`, or `member`). |
A key is bound to exactly one organization, fixed when it is issued. That
organization decides which shared memory, rules, vaults, and conversations the
key can reach. A login is not always bound to the same organization as your
previous one, so show `org.name` to the person after every login, and read
`org.role` rather than inferring authority from anything else.
## Key format
| Prefix | What it is |
| --- | --- |
| `cxy_live_` | A user key, either kind: what you send as a Bearer token. |
| `cxy_svc_` | A service token, minted by an operator. It is not a user key, and an endpoint that expects one rejects the other with `401`. |
Treat everything after the prefix as opaque.
## Verify the key
`GET /api/v1/me` echoes the identity behind a key. Call it once after you store
a new key to confirm it works end to end.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/me \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/me', {
headers: { Authorization: `Bearer ${process.env.CRUXY_API_KEY}` },
})
if (!res.ok) throw new Error(`key check failed: ${res.status}`)
const me = await res.json()
console.log(`key ${me.api_key_id} belongs to user ${me.user_id}`)
```
```json
{
"api_key_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"user_id": "6f9c1e2a-1b2c-4d5e-8f90-1a2b3c4d5e6f"
}
```
A `200` means the key works. A `401` `unauthorized` means it is missing,
malformed, unknown, expired, or revoked: the same answer for each. Check the
`Authorization` header, then the key itself, then its `expires_at`.
## Expiry and revocation
- **A signed-in credential stops working at `expires_at`,** 90 days after
issue. The time is fixed when the key is issued: use does not extend it, and
there is no refresh grant. Prompt for a new device login before it passes.
- **A developer API key does not expire.** Rotate or revoke it in the Console.
- **You cannot revoke a signed-in credential yourself today.** Its holder can
neither rotate nor revoke it, so it ends only at `expires_at`, when its
organization membership ends, or when Cruxy support revokes it. If one leaks,
ask Cruxy support to revoke it.
- **An expired, revoked, or unknown key is `401` `unauthorized`,** with
`WWW-Authenticate: Bearer`. A key whose owner's account is suspended is `403`
`account_suspended`.
- **Revocation applies from the next request,** since nothing is cached. A
stream already running is not cut.
## What a key is allowed to do
A key can carry restrictions. A request outside them is refused:
| Restriction | Refusal |
| --- | --- |
| Allowed scopes: the endpoint families the key may call, and whether it may use memory, rules, or files | `403` `key_scope_forbidden`, naming `required_scope` when a request field asked for a family the key lacks |
| Model allowlist: the tiers the key may call | `403` `model_not_allowed` |
| Read-only: the key may not perform billable operations | `403` `key_read_only` |
| Monthly spend cap | `429` `key_spend_cap_exceeded` |
A `403` never clears by waiting; re-scope the key or issue a new one. For a
developer API key, the Console sets the model allowlist and the monthly spend
cap.
Memory recall, memory capture, and your organization's standing rules are on by
default for a signed-in credential, which includes every key device login
mints, and off by default for a developer API key. To get the same behavior from
any key, set `memory` and `rules` on each chat request; see
[Memory](/docs/en/build/memory) and [Standing rules](/docs/en/build/rules).
## Keep the key safe
- **Treat it like a password.** Anyone who holds it can call the API as you.
- **Keep it on the server.** Never ship a key to a browser. Put a server route
between the browser and the gateway that adds the key; see
[Streaming chat in the browser](/docs/en/resources/cookbook/streaming-ui).
- **Read it from the environment.** Keep it in an environment variable or a
secrets manager, never in source control. Every sample on this site reads
`CRUXY_API_KEY`.
- **Use a developer API key on a server.** A signed-in credential carries its
owner's memory and rules into every request, expires, and cannot be revoked
by its holder.
- **Store it the moment it is issued.** The token endpoint returns it once; if
you lose it, run device login again for a new key.
- **Log the `key_id`, never the key.** `key_id` and `expires_at` identify a key
and tell you when to replace it.
- **Use one key per environment.** Separate keys for local, staging, and
production keep each `key_id` meaningful in your logs.
- **Issue the narrowest key a service needs.** A key whose allowed scopes omit
memory can never read or write anyone's memory through chat.
- **Send it only to `https://api.cruxy.ai`.**
## Next steps
- [Authentication](/docs/en/get-started/authentication): send the key on every request.
- [Quickstart](/docs/en/get-started/quickstart): make your first chat call.
- [Errors and retries](/docs/en/build/errors-and-retries): handle `401`, `403`, and every other error code.
================================================================================
Page: Authentication
URL: https://learn.getcruxy.com/docs/en/get-started/authentication
Markdown: https://learn.getcruxy.com/docs/en/get-started/authentication.md
================================================================================
# Authentication
Send your Cruxy API key as a Bearer token in the Authorization header on every authenticated request.
Every authenticated endpoint reads one header: `Authorization: Bearer`
followed by your `cxy_live_` key. To mint a key, confirm it works, and store it
safely, see [Get your API key](/docs/en/get-started/api-keys).
## The Authorization header
Send the key on every authenticated request:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/me \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/me', {
headers: { Authorization: `Bearer ${process.env.CRUXY_API_KEY}` },
})
console.log(res.status, await res.json())
```
The same key and header work on the OpenAI-compatible routes,
`POST /v1/chat/completions` and `GET /v1/models`: point an OpenAI client at the
base URL `https://api.cruxy.ai/v1` and give it your Cruxy key as its API key.
[Migrate from OpenAI](/docs/en/migrate/from-openai) covers the rest of the
switch.
## Endpoints that take no key
These endpoints are public:
| Endpoint | Why it is public |
| --- | --- |
| `POST /api/v1/device/code`, `POST /api/v1/device/token` | Device login issues keys. |
| `POST /api/v1/auth/oauth/start`, `POST /api/v1/auth/oauth/token` | Native sign-in issues keys. |
| `GET /api/status`, `GET /api/health`, `GET /api/ready` | Platform status and health probes. |
## When the key is rejected
A `401` `unauthorized` means the key is missing, malformed, unknown, expired,
or revoked: the same answer for each. A signed-in key stops working at its
`expires_at`; see [Expiry and revocation](/docs/en/get-started/api-keys#expiry-and-revocation).
Check the `Authorization` header, then
[verify the key](/docs/en/get-started/api-keys#verify-the-key).
A `403` means the key is valid but its authority does not cover the call:
| `code` | Meaning |
| --- | --- |
| `key_scope_forbidden` | The key is not scoped for this endpoint family, or for the memory, rules, or files feature a request field asked for. |
| `model_not_allowed` | The key's model allowlist excludes the requested tier. |
| `key_read_only` | The key may not perform billable operations. |
A `403` never clears by waiting: re-scope the key or issue a new one.
[Errors and retries](/docs/en/build/errors-and-retries) lists every error code.
## Next steps
- [Get your API key](/docs/en/get-started/api-keys): device login, key format, and key hygiene.
- [Chat](/docs/en/build/chat): make an authenticated chat call.
- [API Reference](/docs/en/reference/api): which endpoints require a key.
================================================================================
Page: Migrate from OpenAI
URL: https://learn.getcruxy.com/docs/en/migrate/from-openai
Markdown: https://learn.getcruxy.com/docs/en/migrate/from-openai.md
================================================================================
# Migrate from OpenAI
Move an OpenAI client to Cruxy: change the base URL and the key, pass a tier as the model, and know what carries over and what differs.
Cruxy serves OpenAI-compatible chat completions at `POST /v1/chat/completions`
and an OpenAI-compatible model list at `GET /v1/models`. Both take OpenAI's
request and return OpenAI's response, so code written for the official `openai` npm package moves
over with three changes: the base URL, the API key, and `model`.
## Change three settings
| Setting | Set it to |
| --- | --- |
| Base URL | `https://api.cruxy.ai/v1` |
| API key | Your Cruxy API key, read from `CRUXY_API_KEY`. It travels as `Authorization: Bearer `, as on OpenAI. |
| `model` | A Cruxy tier, `kavi`, `vaani`, or `mira`, or `auto` to let the router pick one. |
The key needs the `chat` endpoint family, the same as for
`POST /api/v1/chat`; a key without it is `403`. On a server, use a developer
API key from the Cruxy Console:
[Get your API key](/docs/en/get-started/api-keys) covers the kinds of key and
how to create one. Authentication, rate limits, and billing work as they do on
`POST /api/v1/chat`.
## Before and after
The same request, first to OpenAI, then to Cruxy. Only the base URL, the key,
and `model` change; the request body and the response you read stay the same.
```bash title="Before (curl)"
# OPENAI_MODEL is the OpenAI model id you call today.
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "'"$OPENAI_MODEL"'",
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```bash title="After (curl)"
curl https://api.cruxy.ai/v1/chat/completions \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```ts title="Before (TypeScript)"
import OpenAI from 'openai'
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
const completion = await client.chat.completions.create({
// The OpenAI model id you call today.
model: process.env.OPENAI_MODEL ?? '',
messages: [{ role: 'user', content: 'Hello' }],
})
console.log(completion.choices[0].message.content)
```
```ts title="After (TypeScript)"
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.CRUXY_API_KEY,
baseURL: 'https://api.cruxy.ai/v1',
})
const completion = await client.chat.completions.create({
model: 'kavi',
messages: [{ role: 'user', content: 'Hello' }],
})
console.log(completion.choices[0].message.content)
```
## Pass a tier as model
`model` takes a tier or `auto`. Any other value, an OpenAI model id included,
is `400` `invalid_request`. These docs map no OpenAI model to a tier: the model
behind a tier is never exposed, and it can change while the tier stays the
same. Choose a tier by its profile, then test it on your own prompts.
| `model` | Profile | Default `reasoning_effort` | `max_tokens` cap |
| --- | --- | --- | --- |
| `kavi` | Deep | `high` | 8192 |
| `vaani` | Balanced | `low` | 4096 |
| `mira` | Fast | `none` | 2048 |
| `auto` | The router picks a tier for each request. | Chosen per request | The cap of the tier that serves |
The response's `model` is the tier that served the request. With `auto`, it is
the tier the router chose, which a budget can also have downgraded, so read it
rather than assume it. [Choosing a tier](/docs/en/build/choosing-a-tier)
compares the tiers, and [Auto routing](/docs/en/build/auto-routing) covers what
the router does.
`GET /v1/models` returns OpenAI's list shape,
`{"object": "list", "data": [...]}`, and its ids are the tiers. For each
tier's context window, output cap, tool support, and reasoning defaults, read
`GET /api/v1/models`; see [Models](/docs/en/reference/models).
## What carries over unchanged
- **The request.** `messages` with the `system`, `user`, `assistant`, and
`tool` roles, plus `temperature`, `top_p`, `max_tokens`, `stream`,
`stream_options`, `tools`, `tool_choice`, `response_format`, and
`reasoning_effort`.
- **The response.** A `chat.completion` object with `id`, `object`, `created`,
`model`, `choices[].message`, `choices[].finish_reason` (`stop`,
`tool_calls`, `length`, or `content_filter`), and `usage` with
`prompt_tokens`, `completion_tokens`, and `total_tokens`.
- **Streaming.** `chat.completion.chunk` frames that end with `data: [DONE]`,
and a usage frame only when you set `stream_options.include_usage`.
- **Tool calling.** `tools` and `tool_choice` on the request, `tool_calls` on
the assistant message, and a `tool` message with `tool_call_id` for each
result.
- **Errors.** OpenAI's envelope, `{"error": {"message", "type", "param", "code"}}`,
which the SDK raises as an `APIError`.
## What differs
- **`model` is a tier,** and the response's `model` names the tier that
served.
- **Fields Cruxy does not implement are ignored, not rejected:** `n`, `stop`,
`seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`,
`user`, and the rest. A request that sends them still runs, and they have no
effect.
- **`max_tokens` is clamped to the tier's cap, and thinking spends it.** `kavi`
thinks by default, so a small `max_tokens` on `kavi` can end with
`finish_reason: "length"` and empty content.
- **`reasoning_effort` takes `none`, `low`, `medium`, or `high`.** Any other
value, OpenAI's `minimal` included, is `400` `invalid_request`.
- **A non-streaming call has a deadline:** 90 seconds on `kavi` and `vaani`, 45
on `mira`. Past it, the call is `504` `upstream_timeout`.
- **An error's `code` is a Cruxy code,** and `param` is always `null`.
- **The route carries none of the Cruxy-native chat features:** no memory, no
standing rules, no vaults, no `file_ids`, and no `cruxy` block. For those,
call `POST /api/v1/chat`; see [Chat](/docs/en/build/chat).
- **The request ID is only in a header,** `X-Request-Id`, and the response's
`usage` carries no cost.
- **A tool call's `id` may not start with `call_`.** Treat it as opaque.
[OpenAI SDK compatibility](/docs/en/migrate/openai-sdk-compatibility) covers
each of these parameter by parameter, with streaming, errors, and how the SDK
retries.
## Next steps
- [OpenAI SDK compatibility](/docs/en/migrate/openai-sdk-compatibility): what the route supports, ignores, and rejects.
- [Choosing a tier](/docs/en/build/choosing-a-tier): pick the tier to pass as `model`.
- [Reasoning effort](/docs/en/build/reasoning-effort): keep thinking from spending your `max_tokens`.
- [Known limitations](/docs/en/build/known-limitations): the limits to plan around.
================================================================================
Page: OpenAI SDK compatibility
URL: https://learn.getcruxy.com/docs/en/migrate/openai-sdk-compatibility
Markdown: https://learn.getcruxy.com/docs/en/migrate/openai-sdk-compatibility.md
================================================================================
# OpenAI SDK compatibility
What POST /v1/chat/completions supports, ignores, and rejects from an OpenAI request, and how its streams, tool calls, and errors behave in the openai SDK.
`POST /v1/chat/completions` is the route the official `openai` npm package
calls when you set `baseURL` to `https://api.cruxy.ai/v1`. It runs your
messages on the tier you name and nothing else: no memory, no standing rules,
and no `cruxy` block, so a run is reproducible. This page goes field by field
through what it does with an OpenAI request. To move a client over first, see
[Migrate from OpenAI](/docs/en/migrate/from-openai).
## Request parameters
| Parameter | Status | Behavior |
| --- | --- | --- |
| `model` | Supported | `kavi`, `vaani`, `mira`, or `auto`. Required. |
| `messages` | Supported | Required. Each message has a `role`, one of `system`, `user`, `assistant`, or `tool`, and a string `content`. An assistant message can replay `tool_calls`, and a `tool` message needs `tool_call_id`. |
| `max_tokens` | Supported, clamped | Unset, or above the tier's cap, is clamped to the cap: 8192 on `kavi`, 4096 on `vaani`, 2048 on `mira`. Thinking is generated inside it. |
| `temperature` | Supported | Forwarded upstream unchanged; the gateway adds no bound of its own. Omit it for the model's default. |
| `top_p` | Supported | Forwarded upstream unchanged. `0` is a real value, so sending it is not the same as omitting it. Omit it for the model's default. |
| `stream` | Supported | `false` by default. |
| `stream_options` | Supported | `include_usage: true` adds the usage frame to a stream. Ignored when `stream` is `false`. |
| `reasoning_effort` | Supported | `none`, `low`, `medium`, or `high`. Omit it for the tier's default. |
| `tools` | Supported | Functions the model may call: `type: "function"`, with `function.name`, `description`, and a JSON Schema `parameters`. |
| `tool_choice` | Supported | `"none"`, `"auto"`, `"required"`, or `{"type": "function", "function": {"name": "…"}}` to force one function. |
| `response_format` | Supported | `{"type": "text"}`, `{"type": "json_object"}`, or `{"type": "json_schema", "json_schema": {"name", "schema", "strict"}}`. |
| `n`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`, `user` | Ignored | Accepted and not applied, so a request that sends them still runs. |
| Any other OpenAI field | Ignored | The route ignores the OpenAI fields it does not implement. That includes `max_completion_tokens`: cap the output with `max_tokens`. |
| `file_ids` | Rejected | The file attachments of `POST /api/v1/chat` are refused here. |
`messages` takes the four roles above; OpenAI's `developer` role is not one of
them, so send your instructions as a `system` message.
### Requests the route rejects
These requests fail with a `4xx` and the code shown:
| Request | Status and `code` |
| --- | --- |
| `model` other than `kavi`, `vaani`, `mira`, or `auto` | `400` `invalid_request` |
| `reasoning_effort` other than `none`, `low`, `medium`, or `high`, OpenAI's `minimal` included | `400` `invalid_request` |
| `tool_choice` names a function that is not in `tools`, or is `"required"` with no tools | `400` `invalid_request` |
| A tool's `parameters` schema exceeds a bound on bytes, depth, or nodes | `400` `invalid_tool_schema` |
| A `response_format` JSON schema is malformed, oversized, too deep, or uses an unsupported keyword | `400` `invalid_schema` |
| The input exceeds the tier's context window | `400` `context_overflow` |
| The server-side input moderation gate, when it is enabled, refuses the request's text | `400` `moderation_rejected`, not billed |
| `response_format` on a tier that does not do constrained decoding | `422` `structured_output_unsupported` |
| An `Idempotency-Key` reused with a different body | `409` `idempotency_conflict` |
One more `422` comes after the model call: `structured_output_unsatisfied`
means the output did not satisfy your `json_schema`. That request is billed,
and it may pass on a retry.
## Response fields
With `stream: false`, the response is a standard `chat.completion` object:
| Field | Value |
| --- | --- |
| `id` | The completion id, such as `chatcmpl-8f3a1c2d`. |
| `object` | `chat.completion`. |
| `created` | Unix seconds at which the completion was produced. |
| `model` | The tier that served the request. With `auto`, the tier the router chose. |
| `choices[].message` | The assistant message, with `tool_calls` when the model called tools. |
| `choices[].finish_reason` | `stop`, `tool_calls`, `length`, or `content_filter`. |
| `choices[].logprobs` | Always `null`. The key is there because OpenAI sends it. |
| `usage` | `prompt_tokens`, `completion_tokens`, and `total_tokens`, plus `prompt_tokens_details.cached_tokens` when the prompt cache was involved. |
The response has no `cruxy` block, no `request_id`, and no cost.
## Cruxy extensions in the SDK
Every Cruxy extension this route honors is either a typed SDK parameter or a
header, so you never need an untyped body field:
- **`reasoning_effort`** is OpenAI's own parameter, typed in
`chat.completions.create`, with Cruxy's values: send only `none`, `low`,
`medium`, or `high`, although the SDK's type allows more. `kavi` thinks at
`high` by default, `vaani` at `low`, and `mira` not at all. See
[Reasoning effort](/docs/en/build/reasoning-effort).
- **`Idempotency-Key`** is a request header; pass it in the SDK's per-request
`headers` option. The first success under a key is stored for 24 hours, and
a retry with the same key and body replays it without calling the model or
billing again. Only a success is stored, and only with `stream: false`. The
SDK does not send this header for you. See
[Reliability](/docs/en/build/reliability).
- **`X-Request-Id`** is on every response, and it is the only request ID this
route returns. The SDK exposes it as `_request_id` on a result and as
`requestID` on an error. Send your own, up to 128 characters of
`[A-Za-z0-9_-]`, and the gateway reuses it. See
[Request IDs and metadata](/docs/en/build/request-ids).
```bash title="curl"
# -i prints the response headers, including X-Request-Id.
curl -i https://api.cruxy.ai/v1/chat/completions \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{
"model": "kavi",
"reasoning_effort": "none",
"max_tokens": 256,
"messages": [
{ "role": "user", "content": "In one sentence, what is a monsoon?" }
]
}'
```
```ts title="TypeScript"
import { randomUUID } from 'node:crypto'
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.CRUXY_API_KEY,
baseURL: 'https://api.cruxy.ai/v1',
})
const completion = await client.chat.completions.create(
{
model: 'kavi',
// Thinking off: the whole max_tokens budget goes to the answer.
reasoning_effort: 'none',
max_tokens: 256,
messages: [{ role: 'user', content: 'In one sentence, what is a monsoon?' }],
},
// The SDK resends the same headers when it retries, so a retry after a
// dropped connection replays a stored success instead of billing again.
{ headers: { 'Idempotency-Key': randomUUID() } },
)
console.log(completion.choices[0].message.content)
console.log(completion.model, completion._request_id)
```
### Memory and standing rules
This route does no memory recall, no memory capture, and no standing-rules
injection, and it takes no `vault_id`. There is nothing to pass through the
SDK to turn them on. To use them, call `POST /api/v1/chat` or
`POST /api/v2/chat` with `fetch`: the SDK's `chat.completions` methods call
`/chat/completions` under the base URL, so they cannot reach either route. See
[Memory](/docs/en/build/memory) and [Standing rules](/docs/en/build/rules).
## Tool calls
Tool calling keeps OpenAI's shapes. Declare `tools`, and when `finish_reason`
is `tool_calls`, run each call in `message.tool_calls`, append the assistant
message and one `tool` message per call, and call again.
A tool call's `id` is a string with no documented format. It may not start
with `call_`, even though examples show `call_abc123`. Copy it unchanged into
`tool_call_id`, and never parse, validate, or build one; see
[Tool-call IDs may not use OpenAI's call\_ format](/docs/en/build/known-limitations#tool-call-ids-may-not-use-openais-call_-format).
```ts
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.CRUXY_API_KEY,
baseURL: 'https://api.cruxy.ai/v1',
})
const tools: OpenAI.ChatCompletionTool[] = [
{
type: 'function',
function: {
name: 'get_weather',
description: 'Get the current weather for a city',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
},
]
const messages: OpenAI.ChatCompletionMessageParam[] = [
{ role: 'user', content: 'Is it raining in Pune?' },
]
let completion = await client.chat.completions.create({ model: 'vaani', tools, messages })
while (completion.choices[0].finish_reason === 'tool_calls') {
const message = completion.choices[0].message
messages.push(message)
for (const call of message.tool_calls ?? []) {
if (call.type !== 'function') continue
const args = JSON.parse(call.function.arguments)
// Echo the id exactly as it arrived; do not assume a call_ prefix.
messages.push({
role: 'tool',
tool_call_id: call.id,
content: JSON.stringify({ city: args.city, raining: true }),
})
}
completion = await client.chat.completions.create({ model: 'vaani', tools, messages })
}
console.log(completion.choices[0].message.content)
```
## Streaming
With `stream: true`, the route speaks OpenAI's Server-Sent Events protocol, in
this order:
1. A role primer: the first `chat.completion.chunk`.
2. Content deltas in `choices[0].delta.content`, or tool-call fragments in
`choices[0].delta.tool_calls`, which you assemble per `index`.
3. A frame that carries `finish_reason`.
4. With `stream_options: {"include_usage": true}` only, a usage frame:
`choices` is an empty array and `usage` is complete. Check `choices` before
you index it.
5. `data: [DONE]`.
If the model fails after the stream has opened, the stream ends with a
`data: {"error": {...}}` frame instead of `[DONE]`, so a cut-off answer never
looks finished. The SDK throws that frame as an `APIError` from the loop:
catch it and discard the partial text. The gateway retries a failing model
only while the stream opens, never after a byte has reached you, and a stream
may run for up to five minutes.
```bash title="curl"
# -N turns off buffering so frames print as they arrive.
curl -N https://api.cruxy.ai/v1/chat/completions \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vaani",
"stream": true,
"stream_options": { "include_usage": true },
"messages": [
{ "role": "user", "content": "Write a haiku about monsoon rain." }
]
}'
```
```ts title="TypeScript"
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.CRUXY_API_KEY,
baseURL: 'https://api.cruxy.ai/v1',
})
const stream = await client.chat.completions.create({
model: 'vaani',
stream: true,
stream_options: { include_usage: true },
messages: [{ role: 'user', content: 'Write a haiku about monsoon rain.' }],
})
let text = ''
try {
for await (const chunk of stream) {
// The usage frame has an empty choices array.
if (chunk.choices.length > 0) text += chunk.choices[0].delta.content ?? ''
if (chunk.usage) console.log('usage:', chunk.usage)
}
} catch (err) {
// A failure after the stream opened arrives as a data: {"error": ...} frame,
// and the SDK throws it here. The partial text is not a finished answer.
throw new Error('stream failed before it finished', { cause: err })
}
console.log(text)
```
A stream stores and replays nothing under `Idempotency-Key`. A retry with the
same key and body streams a new generation; when both attempts complete, the
request is billed once. [Streaming](/docs/en/build/streaming) covers the
stream of `POST /api/v1/chat`, which differs: it sends no error frame.
## Errors
Errors on `POST /v1/chat/completions` and `GET /v1/models` use OpenAI's
envelope:
```json
{
"error": {
"code": "unauthorized",
"message": "unauthorized",
"param": null,
"type": "authentication_error"
}
}
```
- **`code`** is the stable Cruxy error code, the same vocabulary as the rest of
the API. Branch on it.
- **`type`** is a coarse category derived from the HTTP status:
`invalid_request_error`, `authentication_error`, `permission_error`,
`not_found_error`, `rate_limit_error`, or `server_error`.
- **`param`** is always `null`: no error on this route names the field at
fault.
The `/api/v1` routes use Cruxy's own envelope instead. The two differ in where
each value sits:
| Item | `/v1/chat/completions` and `/v1/models` | `/api/v1` routes |
| --- | --- | --- |
| Envelope | `{"error": {"message", "type", "param", "code"}}` | `{"code", "error", "request_id", ...}` |
| The code to branch on | `error.code` | `code` |
| The message | `error.message` | `error` |
| The request ID | The `X-Request-Id` header only | `request_id` in the body |
| Detail fields such as `reset_at`, `mira_available`, or `limit` | Not carried | In the body |
Because this envelope has no place for the detail fields, read the headers
instead: every `429` carries `Retry-After`, and the `X-RateLimit-*` headers
describe the per-minute request window.
[Errors and retries](/docs/en/build/errors-and-retries) lists every code and
which ones to retry.
The SDK throws a non-2xx response as an `APIError`, with the status in
`status`, the Cruxy code in `code`, and `X-Request-Id` in `requestID`:
```bash title="curl"
# OpenAI's "minimal" is not a Cruxy reasoning_effort, so this is a 400.
curl -i https://api.cruxy.ai/v1/chat/completions \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"reasoning_effort": "minimal",
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```ts title="TypeScript"
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.CRUXY_API_KEY,
baseURL: 'https://api.cruxy.ai/v1',
})
try {
await client.chat.completions.create({
model: 'kavi',
// OpenAI's "minimal" is not a Cruxy reasoning_effort, so this is a 400.
reasoning_effort: 'minimal',
messages: [{ role: 'user', content: 'Hello' }],
})
} catch (err) {
if (!(err instanceof OpenAI.APIError)) throw err
// 400 invalid_request, and the X-Request-Id to quote in a support request.
console.error(err.status, err.code, err.requestID)
}
```
### Retries in the SDK
The `openai` package retries a failed request twice by default: after a
connection error, and on `408`, `409`, `429`, and any `5xx`. It waits for
`Retry-After` when that is 60 seconds or less. Against Cruxy, three of those
retries are unlikely to help:
- **`504` `upstream_timeout`.** The gateway made one attempt and does not retry
it, and the same request usually takes as long again. Stream long turns, or
lower `max_tokens` or `reasoning_effort`, rather than rely on the retry.
- **`429` spend and budget codes,** such as `budget_exhausted`. They clear only
at their reset, and their `Retry-After` can be days away, so a retry sooner
fails the same way. Branch on `code` and stop retrying them.
- **`409` `idempotency_conflict`.** The key was reused with a different body,
and the SDK resends that same body. Send the new body under a new key.
A `504` or a `429` is not billed. To make the SDK's retries after a dropped
connection safe, send an `Idempotency-Key`, as the extensions sample does.
## Known differences
These are the places where this route behaves unlike OpenAI's, collected in
one list:
- `model` takes a tier or `auto`, and the response's `model` is the tier that
served, never an upstream model id.
- Fields the route does not implement are ignored without an error, so a
`stop` sequence or `n` has no effect.
- Thinking spends `max_tokens` and is not returned; see
[Kavi's thinking spends max_tokens and isn't returned](/docs/en/build/known-limitations#kavis-thinking-spends-max_tokens-and-isnt-returned).
- A non-streaming call ends at the tier's deadline with `504`
`upstream_timeout`; see
[Non-streaming calls have a 90-second ceiling](/docs/en/build/known-limitations#non-streaming-calls-have-a-90-second-ceiling).
- A tool call's `id` may not use the `call_` format; see
[Tool-call IDs may not use OpenAI's call\_ format](/docs/en/build/known-limitations#tool-call-ids-may-not-use-openais-call_-format).
- `error.code` is a Cruxy code, `param` is always `null`, and the detail fields
of a Cruxy error do not travel.
- The response carries no request ID in its body and no cost in its `usage`.
- No memory, standing rules, vaults, or file attachments.
[Known limitations](/docs/en/build/known-limitations) lists the limits that
apply across the API, each with a workaround.
## Next steps
- [Migrate from OpenAI](/docs/en/migrate/from-openai): change the base URL, the key, and `model`.
- [Chat](/docs/en/build/chat): the Cruxy-native route, with memory, standing rules, and the `cruxy` block.
- [Errors and retries](/docs/en/build/errors-and-retries): every error code, and which ones to retry.
- [Known limitations](/docs/en/build/known-limitations): the limits to plan around.
================================================================================
Page: Chat
URL: https://learn.getcruxy.com/docs/en/build/chat
Markdown: https://learn.getcruxy.com/docs/en/build/chat.md
================================================================================
# Chat
OpenAI-compatible chat completions across the Cruxy tiers, streaming or not.
`POST /api/v1/chat` runs an OpenAI-compatible chat completion. You pick a
public tier, send a list of messages, and read back either a single JSON
completion or a stream of Server-Sent Events. Here is the smallest
complete call:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'kavi',
stream: false,
messages: [{ role: 'user', content: 'Hello' }],
}),
})
const completion = await res.json()
```
## Tiers, not provider models
`model` is a **public tier**, one of:
| Tier | Profile |
| --- | --- |
| `kavi` | Deep |
| `vaani` | Balanced |
| `mira` | Fast |
The gateway resolves each tier to an upstream provider model on the
server. You never send or receive an upstream provider id. Tiers are the
stable contract, so your code keeps working when the backing model
changes. [Choosing a tier](/docs/en/build/choosing-a-tier) compares them; to
discover the catalog at runtime rather than hardcoding these strings, see
[Models](/docs/en/reference/models).
`model` also accepts `"auto"`, which hands tier selection to the gateway's
router. It resolves to a concrete tier before billing and the upstream call,
and `routed_tier` on the response (see [The cruxy block](#the-cruxy-block))
names the tier that served. [Auto routing](/docs/en/build/auto-routing) covers
what the router guarantees.
## Request options
Beyond `model` and `messages`, the request accepts:
| Field | Type | Example | Purpose |
| --- | --- | --- | --- |
| `stream` | boolean | `true` | Stream SSE chunks. Optional and `false` by default, which returns one JSON object. |
| `temperature` | number | `0.7` | Sampling temperature. |
| `max_tokens` | integer | `1024` | Cap on tokens generated. |
| `reasoning_effort` | string | `"medium"` | How much the model thinks before answering: `none`, `low`, `medium`, or `high` (see [Reasoning effort](/docs/en/build/reasoning-effort)). |
| `tools` | array | | Functions the model may call (see [Define tools](/docs/en/build/tools)). |
| `tool_choice` | string or object | `"auto"` | Whether and which tool the model calls. |
| `memory` | object | `{ "recall": true }` | Memory recall and capture for this request (see [Memory](/docs/en/build/memory)). |
| `rules` | boolean | `false` | Standing-rules injection for this request (see [Standing rules](/docs/en/build/rules)). |
Each `messages` entry has a `role` and `content`. Assistant turns that
call tools carry `tool_calls`, and your replies to those calls carry a
`tool_call_id`; both are covered in
[Handle tool calls](/docs/en/build/tool-calls).
## Streaming vs. not
`stream` controls the response format.
### stream: false
This is what you get when you omit `stream`: a single JSON object. Cruxy
flattens it: the assistant reply is the
top-level `message`, alongside `finish_reason`, `usage`, and the `cruxy`
block. There is no OpenAI `choices[]` array. Read `message.content`
directly, not `choices[0].message`.
```json
{
"id": "chatcmpl-8f3a1c2d",
"model": "kavi",
"message": { "role": "assistant", "content": "Namaste." },
"finish_reason": "stop",
"usage": {
"input_tokens": 12,
"output_tokens": 48,
"billable_input_tokens": 12,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"cost": { "tier": "kavi", "amount": "", "currency": "" }
},
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"routed_tier": "kavi",
"reasoning_effort": "high"
}
}
```
`` and `` are placeholders: a real `amount` is a number. See
[Usage and cost](/docs/en/build/usage-and-cost) for the full `cost` object.
### stream: true
You get an OpenAI-style Server-Sent Events stream: a sequence of
`chat.completion.chunk` events, each on a `data:` line, terminated by a
final `data: [DONE]`. Unlike the non-streaming reply, a chunk uses the
OpenAI `choices[]` shape: each token arrives in `choices[0].delta.content`.
```bash
curl -N https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": true,
"messages": [
{ "role": "user", "content": "Write a haiku about monsoon rain." }
]
}'
```
`data: [DONE]` is the only signal that the completion finished: a stream that
ends without it failed partway. [Streaming](/docs/en/build/streaming) covers
the frame order, the chunk and usage shapes, a complete TypeScript reader, and
how to detect a failed stream.
## The cruxy block
Cruxy-native fields are additive and namespaced under a `cruxy` key. They
never alter the OpenAI-shaped fields, so a vanilla OpenAI client parses the
response unchanged and ignores the extra key. On a chat response the block
carries three fields for the turn:
- `request_id`: a UUID for the call, e.g.
`a1b2c3d4-e5f6-7890-abcd-ef1234567890`. Log it; it identifies the call in
support requests and pairs with [Feedback](/docs/en/build/feedback).
- `routed_tier`: the tier that actually served the request. This is how you
learn which tier ran when you send `model: "auto"`.
- `reasoning_effort`: the effort the call ran at, after any tier default is
applied.
When they apply, the block also reports what the turn recalled, captured, or
injected: `memory_recall`, `memory_capture`, `rules_applied`, `files_omitted`,
and `undeclared_tool_replay`.
[Request IDs and metadata](/docs/en/build/request-ids#the-cruxy-block) lists
every field, and how to quote `request_id` when you report a problem.
Cost and token counts are not in this block; they live in `usage`.
## Usage
Every chat response carries a `usage` object with the token accounting and
the call's `cost`. The token fields depend on the response mode:
non-streaming uses `input_tokens` and `output_tokens`; streaming uses
`prompt_tokens`, `completion_tokens`, and `total_tokens`. Both also carry
`billable_input_tokens`, `cache_read_tokens`, and `cache_write_tokens`.
`usage.cost` breaks down the price of the call:
| Field | Meaning |
| --- | --- |
| `tier` | The public tier that served the request. |
| `amount` | Total cost of the call. |
| `currency` | Currency of `amount`. |
| `input_unit` | Input rate, per million tokens. |
| `output_unit` | Output rate, per million tokens. |
| `cache_read_unit` | `input_unit` × the cache discount, for cached reads. |
Amounts and rates are specific to your account and shown in your Cruxy
Console. [Usage and cost](/docs/en/build/usage-and-cost) covers which tokens
are billed and how a stream reports usage.
## Errors
Chat returns `400` for a malformed request, `401` `unauthorized` for a
missing, expired, or revoked key, `429` when you hit a limit, such as
`rate_limited` or `budget_exhausted`, and `5xx` when the model call fails.
With `stream: false` the gateway makes one attempt, bounded by the tier's
deadline; an attempt that runs past it is `504` `upstream_timeout`, which the
gateway does not retry and does not bill (see
[Timeouts](/docs/en/build/errors-and-retries#timeouts)). See
[Errors](/docs/en/reference/errors) for the response shape,
[Errors and retries](/docs/en/build/errors-and-retries) for every code and
which ones to retry, and [Rate limits and budgets](/docs/en/build/rate-limits)
for how to tell a rate limit from a budget.
## Next steps
- [Streaming](/docs/en/build/streaming): read the SSE stream and detect a failed one.
- [Define tools](/docs/en/build/tools): let the model call your functions.
- [Reasoning effort](/docs/en/build/reasoning-effort): keep thinking from spending your `max_tokens`.
- [Known limitations](/docs/en/build/known-limitations): the limits to plan around, such as the 90-second non-streaming ceiling.
- [Streaming chat in the browser](/docs/en/resources/cookbook/streaming-ui): consume the SSE stream in a UI.
- [Models](/docs/en/reference/models): discover tiers at runtime.
- [Errors and retries](/docs/en/build/errors-and-retries): handle `400`, `401`, and `429`, and retry safely.
================================================================================
Page: Streaming
URL: https://learn.getcruxy.com/docs/en/build/streaming
Markdown: https://learn.getcruxy.com/docs/en/build/streaming.md
================================================================================
# Streaming
Read chat completions as Server-Sent Events: the frame order, the chunk and usage shapes, and how to tell a finished stream from a failed one.
`stream` is optional and `false` by default. With `stream: true`,
`POST /api/v1/chat` answers with `text/event-stream`: OpenAI-style Server-Sent
Events in which every event is one `chat.completion.chunk` JSON object on a
`data:` line. The stream ends with the literal line `data: [DONE]`, and that
line is the only signal that the completion finished.
```bash title="curl"
curl -N https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vaani",
"stream": true,
"messages": [
{ "role": "user", "content": "Write a haiku about monsoon rain." }
]
}' | tee stream.txt
# The completion finished only if the stream ended with [DONE].
grep -q '^data: \[DONE\]' stream.txt || echo "stream failed before [DONE]"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'vaani',
stream: true,
messages: [{ role: 'user', content: 'Write a haiku about monsoon rain.' }],
}),
})
if (!res.ok || !res.body) throw new Error(`chat failed: ${res.status} ${await res.text()}`)
const reader = res.body.getReader()
const decoder = new TextDecoder()
let buffer = ''
let text = ''
let finishReason: string | null = null
let usage: Record | undefined
let sawDone = false
read: while (true) {
const { value, done } = await reader.read()
if (done) break
buffer += decoder.decode(value, { stream: true })
const lines = buffer.split('\n')
buffer = lines.pop() ?? '' // keep a partial line for the next read
for (const raw of lines) {
const line = raw.trim()
if (!line.startsWith('data:')) continue
const data = line.slice('data:'.length).trim()
if (data === '[DONE]') {
sawDone = true
break read
}
const chunk = JSON.parse(data)
if (chunk.usage) usage = chunk.usage
const choice = chunk.choices[0] // undefined on the usage frame
if (!choice) continue
if (choice.delta.content) text += choice.delta.content
if (choice.finish_reason) finishReason = choice.finish_reason
}
}
if (!sawDone) {
// The stream ended early: a failed completion, not a short one.
throw new Error('stream ended before [DONE]; discard the partial output')
}
console.log(text)
console.log(finishReason, usage)
```
## Frame order
A successful stream sends these frames, in this order:
| # | Frame | `choices` | What it carries |
| --- | --- | --- | --- |
| 1 | Role primer | One element; `delta` is `{"role": "assistant", "content": ""}` | The turn's `cruxy` block: `routed_tier`, `request_id`, and `reasoning_effort`, plus `memory_recall` and `rules_applied` when they apply. |
| 2 | Content or tool-call deltas, zero or more | One element; `delta` has a `content` fragment or `tool_calls` fragments, never both | The answer, piece by piece. `finish_reason` is `null`. |
| 3 | Finish | One element; `delta` is `{}` | `finish_reason`. No usage. |
| 4 | Usage | `[]`, an empty array | The final `usage` object, plus `cruxy.memory_capture` when the turn stored memory. |
| 5 | `data: [DONE]` | | Nothing. It is not JSON; do not parse it. |
On the wire:
```text
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}],"cruxy":{"routed_tier":"vaani","request_id":"a1b2c3d4-e5f6-7890-abcd-ef1234567890","reasoning_effort":"low"}}
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"content":"Clouds lean"},"finish_reason":null}]}
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{"content":" on the hills"},"finish_reason":null}]}
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"chatcmpl-8f3a1c2d","object":"chat.completion.chunk","model":"vaani","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":48,"total_tokens":60,"cost":{"amount":,"currency":"","tier":"vaani",...}}}
data: [DONE]
```
Changing the framing or this order is a breaking change that ships only under a
new dated `Cruxy-Version`, so you can rely on it. New `cruxy.*` fields can
appear without a version change, so ignore fields you do not recognize.
## The chunk
| Field | Meaning |
| --- | --- |
| `id` | The completion id. |
| `object` | `chat.completion.chunk`. |
| `model` | The tier. |
| `choices` | One element on every frame that carries output; an empty array on the usage frame. |
| `choices[0].index` | `0`. |
| `choices[0].delta` | The fragment: `role` on the primer, then `content` or `tool_calls`, then `{}` on the finish frame. |
| `choices[0].finish_reason` | `null` until the finish frame, then `stop`, `tool_calls`, `length`, or `content_filter`. |
| `usage` | Present only on the usage frame. |
| `cruxy` | The Cruxy-native block; see [Chat](/docs/en/build/chat#the-cruxy-block). |
> **Warning:** Check that `choices` is non-empty before you read `choices[0]`. The usage frame
> has `choices: []` and arrives on every successful stream, so indexing it
> unguarded crashes every time.
The role primer's `content` is an empty string, not absent. Appending every
`content` you see is safe; the primer adds nothing.
### Tool-call deltas
When the model calls a tool, deltas carry `tool_calls` fragments instead of
`content`. Every fragment has an `index` and a piece of `function.arguments`,
a JSON string; fragments can also carry the call's `id`, `type`, and
`function.name`. Assemble the fragments per `index`, concatenating the
`arguments` pieces. A finish frame with `finish_reason: "tool_calls"` means the
fragments are complete: run the calls and answer them as
[Handle tool calls](/docs/en/build/tool-calls) describes.
## The usage frame
The usage frame is the only frame with `usage`, and it is never merged into the
finish frame before it, because an OpenAI-compatible client reads usage only
from the frame whose `choices` is empty.
| Field | Meaning |
| --- | --- |
| `prompt_tokens` | Input tokens. |
| `completion_tokens` | Output tokens. |
| `total_tokens` | The sum. |
| `billable_input_tokens`, `cache_read_tokens`, `cache_write_tokens` | Omitted when zero; absence means zero. |
| `prompt_tokens_details` | The prompt-cache breakdown, present only when the prompt cache was used. |
| `cost` | An object, not a number: `amount`, `currency`, `tier`, and the per-million-token rates `input_unit`, `output_unit`, and `cache_read_unit`. Read `cost.amount` for a scalar. |
A stream names its token fields the OpenAI way (`prompt_tokens`,
`completion_tokens`); a non-streaming response uses `input_tokens` and
`output_tokens`. [Usage and cost](/docs/en/build/usage-and-cost) covers both
shapes, the `cost` object, and which tokens are billed.
## [DONE] is the only success signal
Once the stream opens, the `200` is already sent. If the completion fails after
that, no error frame follows: the gateway stops sending and closes the
response without the finish frame, the usage frame, or `data: [DONE]`.
> **Important:** A stream that ends before `data: [DONE]`, whether the body closes or a read
> returns end-of-stream, is a failed, incomplete completion. Discard the partial
> output or retry; never present it as a finished answer. The tokens already
> delivered are real and are billed as a partial usage row.
To detect it on the client:
1. Set a flag when you read `data: [DONE]`, as the sample above does with
`sawDone`.
2. When the read loop ends, treat a missing flag as a failure.
3. Treat an exception from the read itself, such as a dropped connection, as a
failure too.
A stream that ends with `[DONE]` is complete, but the answer can still be cut
short: `finish_reason: "length"` means the output reached `max_tokens` or the
tier's cap. On `kavi`, which reasons by default, a small `max_tokens` can be
spent entirely on thinking; see
[Features overview](/docs/en/get-started/features#three-tiers-and-auto).
The gateway retries a failing model only while the stream opens, never after a
byte has reached you. A stream may run for up to five minutes.
A stream stores and replays nothing under `Idempotency-Key`, since a stream
cannot be replayed: a retry with the same key and body streams a new
generation. When both attempts complete, the request is billed once. An attempt
that did not complete is billed for the tokens it delivered, and the retry is
billed as well.
## Other chat wires
The rules above are for `POST /api/v1/chat`. The other two chat endpoints end
their streams differently:
- `POST /api/v2/chat` sends named events (`start`, `reasoning`, `delta`) and
always ends with exactly one `finish` or `error` frame. It has no `[DONE]`;
a stream that closes without `finish` or `error` failed.
- `POST /v1/chat/completions` ends a stream that fails after it opens with a
`data: {"error": {...}}` frame instead of `[DONE]`. It sends the usage frame
only when the request sets `stream_options: {"include_usage": true}`.
To render a stream in a browser without exposing your key, see
[Streaming chat in the browser](/docs/en/resources/cookbook/streaming-ui).
## Next steps
- [Chat](/docs/en/build/chat): the request options and the non-streaming response.
- [Reliability](/docs/en/build/reliability): retry a stream that failed, and the deadlines it avoids.
- [Handle tool calls](/docs/en/build/tool-calls): answer the tool calls a stream returns.
- [Errors and retries](/docs/en/build/errors-and-retries): what to retry, and how.
================================================================================
Page: Embeddings
URL: https://learn.getcruxy.com/docs/en/build/embeddings
Markdown: https://learn.getcruxy.com/docs/en/build/embeddings.md
================================================================================
# Embeddings
Turn text into vectors with the cruxy-embed model.
`POST /api/v1/embeddings` takes one string or an array and returns a
vector per input, ready to store and compare for search, clustering, or
retrieval. It is OpenAI-compatible. Here is a single-string embedding:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/embeddings \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cruxy-embed",
"input": "The quick brown fox"
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/embeddings', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'cruxy-embed',
input: 'The quick brown fox',
}),
})
const { data } = await res.json()
const vector = data[0].embedding // number[]
```
## The model
`model` must be `"cruxy-embed"`. It is the only embedding model on the
gateway.
## Single string or array
`input` accepts either a single string or an array of strings. Pass an
array to embed a batch in one call:
```bash
curl https://api.cruxy.ai/api/v1/embeddings \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cruxy-embed",
"input": ["The quick brown fox", "jumps over the lazy dog"]
}'
```
Each input maps to one entry in `data`, and the entry's `index` matches
the position of its input.
## Response shape
```json
{
"object": "list",
"model": "cruxy-embed",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0023, -0.0117]
}
],
"usage": {
"prompt_tokens": 8,
"total_tokens": 8
},
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"cost": { "tier": "cruxy-embed" }
}
}
```
The `embedding` array is shown truncated; a real vector holds many more
floats. The `cost` object carries the same fields documented under
[Chat](/docs/en/build/chat#usage).
| Field | Meaning |
| --- | --- |
| `object` | Always `"list"`. |
| `data` | One `embedding` entry per input, each with its `index` and `embedding` vector. |
| `usage` | `prompt_tokens` and `total_tokens` for the request. |
| `cruxy` | Additive block with the `request_id` and `cost` (see [Chat](/docs/en/build/chat#the-cruxy-block)). |
The `embedding` array holds the vector as floating-point numbers. Read a
result at `data[i].embedding`, matching `data[i].index` to your input's
position.
## Errors
Embeddings returns `400` for a malformed request, `401` for a bad key,
and `429` for `rate_limited` or `budget_exhausted`. See
[Errors](/docs/en/reference/errors).
## Next steps
- [Retrieval-augmented generation](/docs/en/resources/cookbook/rag): index a corpus and answer from it, end to end.
- [Rerank](/docs/en/build/rerank): order retrieved candidates against a query.
- [Chat](/docs/en/build/chat): completions across the tiers.
- [Errors](/docs/en/reference/errors): the error shape.
================================================================================
Page: Rerank
URL: https://learn.getcruxy.com/docs/en/build/rerank
Markdown: https://learn.getcruxy.com/docs/en/build/rerank.md
================================================================================
# Rerank
Order candidate documents against a query with the cruxy-rerank model.
`POST /api/v1/rerank` scores each candidate document against your query
and returns them best-first, so you can pick the strongest matches from a
shortlist. Here is a working request:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/rerank \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cruxy-rerank",
"query": "What is the capital of France?",
"documents": [
"Paris is the capital and largest city of France.",
"Berlin is the capital of Germany.",
"The Louvre is a museum in Paris.",
"France is in Western Europe."
],
"top_n": 3
}'
```
```ts title="TypeScript"
const documents = [
'Paris is the capital and largest city of France.',
'Berlin is the capital of Germany.',
'The Louvre is a museum in Paris.',
'France is in Western Europe.',
]
const res = await fetch('https://api.cruxy.ai/api/v1/rerank', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'cruxy-rerank',
query: 'What is the capital of France?',
documents,
top_n: 3,
}),
})
const { results } = await res.json()
const top = results[0] // highest relevance_score
const topDocument = documents[top.index]
```
## The model
`model` must be `"cruxy-rerank"`. It is the only rerank model on the
gateway.
## Not OpenAI-compatible
Most of the gateway mirrors the OpenAI shape. This endpoint does not;
it follows a Cohere-style rerank contract instead. There is no `object`
envelope, and the results carry **indices into your input**, not the
document text. Map each `index` back to your own `documents` array to
recover the text, as the TypeScript example does above.
## Request fields
| Field | Type | Example | Purpose |
| --- | --- | --- | --- |
| `model` | string | `"cruxy-rerank"` | Must be `"cruxy-rerank"`. |
| `query` | string | `"What is the capital of France?"` | Scored against each document. |
| `documents` | array | | Candidate strings to rank. |
| `top_n` | integer | `3` | Cap on how many results come back. Omit to rank every document. |
## Ranking and top_n
Every document is scored against `query`, and the response is sorted by
`relevance_score` in **descending** order, the most relevant document
first. `top_n` caps how many results return; with `top_n: 3` over four
documents, only the top three come back. Omit `top_n` to get all of
them, still ranked.
## Response shape
```json
{
"model": "cruxy-rerank",
"results": [
{ "index": 0, "relevance_score": 0.98 },
{ "index": 2, "relevance_score": 0.71 },
{ "index": 3, "relevance_score": 0.12 }
],
"usage": {
"prompt_tokens": 8,
"total_tokens": 8
},
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"cost": { "tier": "cruxy-rerank" }
}
}
```
| Field | Meaning |
| --- | --- |
| `model` | Always `"cruxy-rerank"`. |
| `results` | Ranked entries, `relevance_score` descending, capped to `top_n`. |
| `index` | Position of the document in the request `documents` array. |
| `relevance_score` | Relevance of that document to `query`; higher is more relevant. |
| `usage` | `prompt_tokens` and `total_tokens` for the request. |
| `cruxy` | Additive block with the `request_id` and `cost` (see [Chat](/docs/en/build/chat#the-cruxy-block)). |
Each result's `index` points back into the `documents` you sent, so read
the ranked text as `documents[results[i].index]`. A dropped document
(one below the `top_n` cut) has no entry.
## Where it fits
Rerank and [Embeddings](/docs/en/build/embeddings) are the two retrieval
primitives on the gateway, and they work at different stages. A common
pattern is to retrieve broadly with embeddings, then rerank the top
candidates to settle their final order.
| | Embeddings | Rerank |
| --- | --- | --- |
| Model id | `cruxy-embed` | `cruxy-rerank` |
| What it does | Turns text into vectors you compare with cosine similarity. | Scores each candidate document against the query directly. |
| Best for | A first-pass search over a large corpus. | Ordering a shortlist before you use it. |
| Learn more | [Embeddings](/docs/en/build/embeddings) | [Retrieval-augmented generation](/docs/en/resources/cookbook/rag) |
## Errors
Rerank returns `400` for a malformed request, `401` for a bad key, and
`429` for `rate_limited` or `budget_exhausted`. See
[Errors](/docs/en/reference/errors).
## Next steps
- [Retrieval-augmented generation](/docs/en/resources/cookbook/rag): the embeddings-to-chat recipe this endpoint sits in.
- [Embeddings](/docs/en/build/embeddings): the vectors you retrieve candidates with.
- [Models](/docs/en/reference/models): discover tiers at runtime.
- [Errors](/docs/en/reference/errors): the error shape.
================================================================================
Page: Moderations
URL: https://learn.getcruxy.com/docs/en/build/moderations
Markdown: https://learn.getcruxy.com/docs/en/build/moderations.md
================================================================================
# Moderations
Classify text against content-safety categories with the cruxy-guard model.
`POST /api/v1/moderations` checks text against the content-safety
categories and returns whether each one matched, plus an overall
`flagged`. It is OpenAI-compatible. Here is a single-string check:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/moderations \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cruxy-guard",
"input": "I want to learn how to bake bread."
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/moderations', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'cruxy-guard',
input: 'I want to learn how to bake bread.',
}),
})
const { results } = await res.json()
if (results[0].flagged) {
// handle flagged content
}
```
## The model
`model` must be `"cruxy-guard"`. It is the only moderation model on the
gateway.
## Single string or array
`input` accepts a single string or an array of strings. With an array,
`results` holds one entry per input, in order.
## Response shape
```json
{
"id": "modr-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"model": "cruxy-guard",
"results": [
{
"flagged": false,
"categories": { "example_category": false },
"category_scores": { "example_category": 0.0 }
}
]
}
```
| Field | Meaning |
| --- | --- |
| `flagged` | `true` if any category matched. Check this first. |
| `categories` | Map of category name to a boolean: whether that category matched. |
| `category_scores` | Map of category name to a score. See the caveat below. |
`categories` and `category_scores` share the same keys, one per
moderation category. The keys shown above are illustrative; treat the
category set as whatever the response returns rather than a fixed list.
## category_scores is binary
The scores are **not** calibrated probabilities. The guard model emits
labels, not likelihoods, so each value in `category_scores` is either
`1.0` or `0.0`, the same information as the boolean in `categories`,
expressed as a number. Do not threshold these scores or read them as
confidence. To decide whether to act, use `flagged` (or the per-category
booleans in `categories`). To run this check in front of a chat call, see
the [Moderation gate](/docs/en/resources/cookbook/moderation-gate) recipe.
## Errors
Moderations returns `400` for a malformed request, `401` for a bad key,
and `429` for `rate_limited` or `budget_exhausted`. See
[Errors](/docs/en/reference/errors).
## Next steps
- [Moderation gate](/docs/en/resources/cookbook/moderation-gate): screen input before it reaches chat.
- [Chat](/docs/en/build/chat): completions across the tiers.
- [Errors](/docs/en/reference/errors): the error shape.
================================================================================
Page: Feedback
URL: https://learn.getcruxy.com/docs/en/build/feedback
Markdown: https://learn.getcruxy.com/docs/en/build/feedback.md
================================================================================
# Feedback
Report the outcome of a prior call: accepts, rejects, thumbs, test results.
`POST /api/v1/feedback` records how an earlier call turned out (an edit
accepted or rejected, a thumbs rating, or a generated test's result),
attached to that call by its `request_id`. Here is a working request:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/feedback \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"type": "thumbs",
"consent": true,
"payload": { "value": "up" }
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/feedback', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
request_id: 'a1b2c3d4-e5f6-7890-abcd-ef1234567890',
type: 'thumbs',
consent: true,
payload: { value: 'up' },
}),
})
const { stored } = await res.json() // 202 Accepted; see below
```
## Joining to a prior call
`request_id` ties the feedback to the call it is about. It is the same
UUID the gateway returns in the `cruxy.request_id` field of a chat,
embeddings, rerank, or moderations response; see
[the cruxy block](/docs/en/build/chat#the-cruxy-block). Log that id when you make
a call, then send it back here when you learn the outcome.
## Event types
`type` must be one of:
| Value | |
| --- | --- |
| `edit_accepted` | |
| `edit_rejected` | |
| `thumbs` | |
| `test_result` | |
Each `type` fixes the shape of `payload`, and the gateway validates it,
rejecting unknown keys, wrong types, missing required fields, and bad enum
values with a `400`. `edit_accepted` and `edit_rejected` share one shape;
`thumbs` and `test_result` each have their own. The
[API Reference](/docs/en/reference/api) carries the exact fields for each;
the request above sends a `thumbs` payload.
## Request fields
| Field | Type | Example | Purpose |
| --- | --- | --- | --- |
| `request_id` | string | `"a1b2c3d4-…"` | The prior call this feedback is about. |
| `type` | string | `"thumbs"` | One of the values above. |
| `consent` | boolean | `true` | Whether this event may be stored. See [What gets stored](#what-gets-stored). |
| `payload` | object | | Event data; its shape is fixed by `type` and validated (see [Event types](#event-types)). |
## What gets stored
Feedback is privacy-gated, and the gate is strict. Two things decide
whether an event is kept:
- `consent`: the per-event boolean you send.
- The account's opt-out: a setting outside this request.
The outcomes are:
- **Not consented, or the account has opted out**: the event is
dropped entirely. No row is written and the payload is never stored.
The response has `stored: false`.
- **Consented, and not opted out**: a row is written, but its payload
is redacted. The event is recorded; the payload you sent is not kept
verbatim.
The payload you send is never persisted as-is. A dropped event keeps
nothing; a stored event keeps everything but the payload. Do not rely on
reading the raw payload back later.
## The response
The endpoint **always returns `202 Accepted`**, for a stored event and
for a dropped one alike. A `202` means the event was accepted for
processing, not that it was stored, so read `stored` to tell the two
apart.
```json
{
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"stored": true
}
```
| Field | Meaning |
| --- | --- |
| `stored` | `true` if a row was written (payload redacted), `false` if the event was dropped. |
| `id` | Identifier for the stored event. |
## Errors
Feedback returns `400` for a malformed request and `401` for a bad key.
A privacy drop is **not** an error; it still returns `202` with
`stored: false`. See [Errors](/docs/en/reference/errors).
## Next steps
- [Chat](/docs/en/build/chat#the-cruxy-block): where `request_id` comes from.
- [Errors](/docs/en/reference/errors): the error shape.
================================================================================
Page: Choosing a tier
URL: https://learn.getcruxy.com/docs/en/build/choosing-a-tier
Markdown: https://learn.getcruxy.com/docs/en/build/choosing-a-tier.md
================================================================================
# Choosing a tier
Compare kavi, vaani, and mira by profile, output cap, deadline, and reasoning default, then pick the tier that fits each call.
You choose a tier on every chat request with the `model` field. A tier is a
stable public id: the gateway resolves it to an upstream model on the server
and never exposes which one, so the tier is the contract your code depends on.
`GET /api/v1/models` returns the live catalog, so read it at runtime instead of
hardcoding what this page shows:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/models \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/models', {
headers: { Authorization: `Bearer ${process.env.CRUXY_API_KEY}` },
})
const { data } = await res.json()
for (const tier of data) {
console.log(tier.id, tier.label, tier.profile, tier.max_output, tier.reasoning)
}
```
## The three tiers
Each tier has a display name (`label`) and a one-word `profile` that names its
role. The profile is a qualitative descriptor, not a performance number.
| Tier | Label | Profile | Output cap | Non-streaming deadline |
| --- | --- | --- | --- | --- |
| `kavi` | Kavi | Deep | 8192 tokens | 90 s |
| `vaani` | Vaani | Balanced | 4096 tokens | 90 s |
| `mira` | Mira | Fast | 2048 tokens | 45 s |
- **Output cap** is the tier's `max_output`. A `max_tokens` you send above it,
or no `max_tokens` at all, is clamped to the cap, so the gateway never
generates more than the cap.
- **Non-streaming deadline** bounds a `stream: false` call. The gateway makes
one attempt; a call that runs past the deadline is cancelled and answered
`504` `upstream_timeout`, and nothing is billed. A streaming call may run for
up to five minutes.
Every tier reads images today (`supports_vision` is `true` on each). Read each
tier's `context_window`, `latency_target_ms`, `supports_tools`, and `pricing`
from the catalog response.
## How each tier reasons
A tier can think before it answers. Each catalog entry's `reasoning` object
gives the effort a request gets when it sends no `reasoning_effort`
(`default_effort`) and whether `low`, `medium`, and `high` are different depths
on that tier (`graded`):
| Tier | Profile | `default_effort` | `graded` | What that means |
| --- | --- | --- | --- | --- |
| `kavi` | Deep | `high` | `false` | Thinks unless you send `none`. The control is on or off, so `low` thinks as long as `high`. |
| `vaani` | Balanced | `low` | `true` | Thinks lightly by default; `medium` and `high` think longer. |
| `mira` | Fast | `none` | `true` | Answers without thinking unless you ask; `low`, `medium`, and `high` are graded. |
These are the shipped defaults; the catalog reports the live ones.
[Reasoning effort](/docs/en/build/reasoning-effort) covers every value and when
to change it.
> **Warning:** `kavi` thinks by default, and its thinking spends the same `max_tokens` budget
> as the answer without coming back as content. A small `max_tokens` on `kavi`
> can run out before any answer is written: the turn ends with
> `finish_reason: "length"` and empty content. Send `reasoning_effort: "none"`
> for a short, direct answer, or leave `max_tokens` unset to keep the tier's cap.
## Pick a tier for the job
| | Kavi | Vaani | Mira |
| --- | --- | --- | --- |
| Model id | `kavi` | `vaani` | `mira` |
| Reach for it when | The task needs multi-step reasoning or a long answer. It thinks by default and has the largest output cap. | Everyday calls. It thinks lightly by default, and its graded effort gives you more depth when a request needs it. | Short, direct calls. It answers without thinking unless you ask. |
| Watch for | Thinking spends `max_tokens`. Leave `max_tokens` unset or large, or send `reasoning_effort: "none"` for a direct answer. | A 4096-token output cap. An answer that needs more ends with `finish_reason: "length"`. | The smallest output cap and a 45 s non-streaming deadline: stream a long turn. On a signed-in plan, `mira` has a monthly request cap of its own. |
[Rate limits and budgets](/docs/en/build/rate-limits) shows how to read the
`mira` request cap and the other budgets in `GET /api/v1/limits`, where it sits
in the signed-in plan's `token_pool`.
## Pin a tier or let the router choose
Name a tier when you need its behavior to hold on every call: the same output
cap, deadline, and reasoning default. Send `model: "auto"` when your requests
vary and you would rather the gateway pick per request;
[Auto routing](/docs/en/build/auto-routing) covers what the router guarantees
and how to read the tier it chose.
A developer API key can carry a model allowlist. A request for a tier the
allowlist excludes is `403` `model_not_allowed`; see
[Get your API key](/docs/en/get-started/api-keys#what-a-key-is-allowed-to-do).
## Next steps
- [Auto routing](/docs/en/build/auto-routing): let the gateway pick the tier per request.
- [Reasoning effort](/docs/en/build/reasoning-effort): control how much a tier thinks.
- [Models](/docs/en/reference/models): every field in the catalog response.
- [Chat](/docs/en/build/chat): send your first request to a tier.
================================================================================
Page: Auto routing
URL: https://learn.getcruxy.com/docs/en/build/auto-routing
Markdown: https://learn.getcruxy.com/docs/en/build/auto-routing.md
================================================================================
# Auto routing
Send model auto and the gateway picks a tier for each request before billing; read cruxy.routed_tier to see which tier served it.
Send `model: "auto"` instead of a tier id and the gateway's server-side router
picks the tier from the request content. It resolves `auto` to `kavi`, `vaani`,
or `mira` before billing and before the upstream call, so `auto` never reaches
a model, and every response names the tier that served it.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"stream": false,
"messages": [
{ "role": "user", "content": "Summarize the causes of the monsoon in three sentences." }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'auto',
stream: false,
messages: [
{ role: 'user', content: 'Summarize the causes of the monsoon in three sentences.' },
],
}),
})
const completion = await res.json()
console.log(completion.cruxy.routed_tier) // the tier that served the call
console.log(completion.cruxy.reasoning_effort) // the effort it ran at
console.log(completion.message.content)
```
## Read the tier that served
The routed tier comes back on every response, in a field that depends on the
endpoint:
| Endpoint | Where the routed tier is |
| --- | --- |
| `POST /api/v1/chat` | `cruxy.routed_tier`. On a stream it rides the role primer, the first frame, before any model output. |
| `POST /api/v2/chat` | `routing.tier`, with `routing.mode` set to `explicit`, `auto`, or `auto_degraded`. On a stream it rides the `start` frame. |
| `POST /v1/chat/completions` | The response's `model`. |
`usage.cost.tier` also names the tier that served the request. The router can
choose differently for two similar requests, so log `routed_tier` with each
`request_id` and branch on the value you read, never on the tier you expected.
```json
{
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"routed_tier": "kavi",
"reasoning_effort": "high"
}
}
```
## What auto does for you
- **Picks a healthy tier.** The router chooses a tier that is available before
the call is sent. Once sent, a call never moves to another tier. When every
tier is unavailable, the request is `503` `all_models_unavailable`; retry it
with backoff.
- **Chooses the reasoning effort.** With `reasoning_effort` omitted, `auto`
chooses the effort per request and echoes it as `cruxy.reasoning_effort`
(`routing.reasoning_effort` on the native wire). Send `reasoning_effort` to
set it yourself.
- **Routes images to a tier that reads them.** A turn whose messages attach an
image goes to a tier with `supports_vision`. Every tier reads images today.
- **Moves to `mira` instead of failing a budget.** When a budget would refuse
the request and `mira` can serve it, the gateway moves an `auto` request to
`mira` rather than answering `429` `budget_exhausted`. On the native wire,
`routing.mode` is `auto_degraded`.
[Rate limits and budgets](/docs/en/build/rate-limits) explains the budgets.
## Size max_tokens for any tier
`max_tokens` is clamped to the cap of the tier the router picks: 8192 on
`kavi`, 4096 on `vaani`, and 2048 on `mira`. If you set a small `max_tokens` on
an `auto` request, also send `reasoning_effort`. Otherwise the router may pick
`kavi`, whose default thinking can spend the whole budget and end the turn with
`finish_reason: "length"` and empty content; see
[Reasoning effort](/docs/en/build/reasoning-effort).
## When to name a tier instead
- **You need the same behavior on every call:** one output cap, one
non-streaming deadline, one reasoning default.
- **You are measuring.** An evaluation or benchmark compares like with like,
so pin the tier it measures.
- **You count tokens first.** `POST /api/v1/tokens/count` rejects `auto` with
`400`; count against the tier you plan to send.
[Choosing a tier](/docs/en/build/choosing-a-tier) compares the three tiers.
## Next steps
- [Choosing a tier](/docs/en/build/choosing-a-tier): what each tier is for.
- [Reasoning effort](/docs/en/build/reasoning-effort): set the effort instead of letting `auto` choose.
- [Streaming](/docs/en/build/streaming): read the routed tier from the role primer.
- [Errors and retries](/docs/en/build/errors-and-retries): retry `503` `all_models_unavailable`.
================================================================================
Page: Reasoning effort
URL: https://learn.getcruxy.com/docs/en/build/reasoning-effort
Markdown: https://learn.getcruxy.com/docs/en/build/reasoning-effort.md
================================================================================
# Reasoning effort
Set how much a tier thinks before it answers, from none to high, and keep hidden thinking from spending your whole max_tokens budget.
`reasoning_effort` caps the hidden thinking a tier does before it answers. You
send one of four values on a chat request, or omit it and the tier applies its
default. The effort the call ran at comes back on the response as
`cruxy.reasoning_effort`.
This request asks `kavi` for a one-line answer, so it turns thinking off and
keeps the budget small:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"reasoning_effort": "none",
"max_tokens": 256,
"messages": [
{ "role": "user", "content": "Is Mumbai north or south of Chennai? Answer in one sentence." }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'kavi',
stream: false,
reasoning_effort: 'none',
max_tokens: 256,
messages: [
{ role: 'user', content: 'Is Mumbai north or south of Chennai? Answer in one sentence.' },
],
}),
})
const completion = await res.json()
console.log(completion.cruxy.reasoning_effort) // "none"
console.log(completion.message.content)
```
## The four values
| Value | Effect |
| --- | --- |
| `none` | Turns thinking off. The tier answers directly. |
| `low`, `medium`, `high` | Turn thinking on. On a graded tier the three are different depths, from `low` to `high`; on an on-or-off tier each one turns thinking on. |
Any other value is `400`. The field is accepted on `POST /api/v1/chat`,
`POST /api/v2/chat`, and `POST /v1/chat/completions`.
## Defaults per tier
Omit `reasoning_effort` and each tier applies its own default. `graded` says
whether `low`, `medium`, and `high` are different depths on the tier. When it
is `false` the control is on or off: `none` turns thinking off, every other
value turns it on, `low` thinks as long as `high`, and only `max_tokens` bounds
it.
| Tier | Profile | Default | `graded` |
| --- | --- | --- | --- |
| `kavi` | Deep | `high` | `false`: on or off |
| `vaani` | Balanced | `low` | `true` |
| `mira` | Fast | `none` | `true` |
With `model: "auto"`, the router chooses the effort per request unless you send
one. These are the shipped defaults; `GET /api/v1/models` reports each tier's
live `reasoning.default_effort` and `reasoning.graded`, read from the same
configuration the chat endpoints apply.
## Thinking spends max_tokens
A tier that thinks generates its thinking inside the same output budget as its
answer. On `POST /api/v1/chat` and `POST /v1/chat/completions`, the thinking is
not returned as content. So a small `max_tokens` on a tier that thinks can be
used up before the answer starts.
> **Warning:** `kavi` thinks by default (`high`), so `kavi` with a small `max_tokens` and no
> `reasoning_effort` can return nothing: the turn ends with
> `finish_reason: "length"` and empty content. For a short, direct answer, send
> `reasoning_effort: "none"`. To
> keep the thinking, leave `max_tokens` unset, which clamps it to the tier's cap,
> or set it large.
Check for the trap when you read a response, so a truncated turn never reaches
your user as an empty answer:
```ts
const { message, finish_reason } = completion
if (finish_reason === 'length' && !message.content) {
// The budget ran out before the answer began. Resend with
// reasoning_effort: 'none', or with max_tokens unset.
}
```
## When to send none
Send `none` when the answer needs no deliberation and you want it fast:
- **Short, direct answers:** a classification, a yes or no, an extracted field,
a one-line reply.
- **A small `max_tokens`:** any time you cap the output tightly, so thinking
cannot consume the cap.
- **A turn that times out:** a `stream: false` call is bounded by the tier's
deadline and answered `504` `upstream_timeout` past it. Lowering
`reasoning_effort` or `max_tokens` shortens the turn; streaming lifts the
limit to five minutes.
Keep thinking on when the answer needs deliberation, and leave `max_tokens`
unset there so the thinking and the answer share the tier's full cap.
## See the thinking
The OpenAI-shaped wires do not return the thinking. The native wire,
`POST /api/v2/chat`, does: a non-streaming response carries a top-level
`reasoning` string, and a stream sends the thinking as `reasoning` events,
separate from the answer's `delta` events. Either way it is present only when
the tier reasoned.
- It is not part of the answer, and it sits outside `message` on purpose. Do
not send it back in the conversation.
- There is no separate reasoning-token count. `usage.output_tokens` already
includes the thinking, undivided.
- The effort is echoed as `routing.reasoning_effort` on the native wire.
## Next steps
- [Choosing a tier](/docs/en/build/choosing-a-tier): each tier's profile, output cap, and default.
- [Auto routing](/docs/en/build/auto-routing): let the router choose the tier and the effort.
- [Streaming](/docs/en/build/streaming): read the stream and its `finish_reason`.
- [Errors and retries](/docs/en/build/errors-and-retries#timeouts): what to do about `504` `upstream_timeout`.
================================================================================
Page: Memory
URL: https://learn.getcruxy.com/docs/en/build/memory
Markdown: https://learn.getcruxy.com/docs/en/build/memory.md
================================================================================
# Memory
Let chat recall stored facts and capture new ones. Memory is on by default for signed-in keys and off for developer keys until a request opts in.
Memory gives chat two independent abilities, each set per request in the
`memory` object. **Recall** reads the stored memories most relevant to the turn
into it as background context. **Capture** lets the gateway store a durable
fact from the turn for later ones. When a request sends no `memory` object, the
key decides whether each one runs.
## Defaults depend on the key
| Key | Recall and capture when `memory` is omitted | `bucket` in `GET /api/v1/limits` |
| --- | --- | --- |
| Signed-in credential: device login, OAuth login, and the Cruxy web app, desktop, CLI, and mobile | On | `subscription` |
| Developer API key, created in the Cruxy Console | Off | `apikey` |
| Headless key | Off | `headless` |
A key minted by device login or OAuth login is a signed-in credential whichever
client ran the flow, a third-party tool included. A developer key's memory is
off because the key often serves people other than its owner: recall would put
the owner's stored memory into their turns, and capture would store what they
say. Use a developer API key for a server; see
[Get your API key](/docs/en/get-started/api-keys#two-kinds-of-key).
## Opt in from a developer key
Send `true` for each side you want. This request comes from an internal tool
that only the key's owner uses, so it recalls the owner's memory and lets the
turn store a new fact:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vaani",
"stream": false,
"memory": { "recall": true, "capture": true },
"messages": [
{ "role": "user", "content": "I work in IST. When should our weekly sync with the London team be?" }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'vaani',
stream: false,
memory: { recall: true, capture: true },
messages: [
{
role: 'user',
content: 'I work in IST. When should our weekly sync with the London team be?',
},
],
}),
})
const completion = await res.json()
console.log(completion.message.content)
console.log(completion.cruxy.memory_recall) // what was recalled, or undefined
console.log(completion.cruxy.memory_capture) // what was stored, or undefined
```
`true` needs a key whose allowed scopes include the `memory` family. An
unscoped key has every family. A key without it gets `403`
`key_scope_forbidden`, refused before anything is billed:
| Error field | Value |
| --- | --- |
| `code` | `key_scope_forbidden` |
| `required_scope` | `memory` |
| `allowed_scopes` | The key's own families, for example `["chat"]` |
A `403` never clears by waiting. A key whose allowed scopes omit `memory` can
never use memory through chat, which makes it the key to issue for a service
that must never see anyone's memory.
> **Warning:** Never store secrets in memory, and never enable capture on anonymous or public
> traffic. Memory has no expiry: a stored value stays, and recall can place it in
> later turns, until someone deletes it. The capture filter drops content that
> carries an email address, a phone number, or a card number; it does not
> promise to catch a password, an API key, or a token. Capture on public traffic
> stores what strangers say into the key owner's memory.
## What recall adds
Recall draws on the same resolved view `GET /api/v1/memories` lists: your
personal memory, your organization's, and every vault you can read whose
`memory_scope` is `default`.
- **`vault_id`** names the vault a chat is inside. Inside a `vault_only` vault,
recall draws on that vault's memory alone, without personal or organization
memory. A `vault_id` you cannot read is treated as if you sent none, never
reported.
- **`mentioned_vault_ids`** adds up to three vaults the turn references, such
as an `@VersionGate` your picker resolved. Mentions share the turn's recall
budget rather than enlarging it, and never make a vault writable.
- **Billing.** The recalled block is billed input, so input tokens fall by its
size when recall is off.
The response reports what was recalled in `cruxy.memory_recall`: the scope and
key of each memory, and for a vault memory its `vault_id` (and `vault_name`
when it resolves). It never includes the stored value, and the field is absent
when nothing was recalled.
```json
{
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"routed_tier": "vaani",
"reasoning_effort": "low",
"memory_recall": {
"count": 1,
"items": [{ "scope": "personal", "key": "timezone" }]
}
}
}
```
## What capture stores
After the turn is answered, capture decides whether the exchange held a durable
fact, such as a stated preference, a stable setting, or a constraint about your
work, and stores it.
- **Where it goes.** Your personal memory, or the vault the request's
`vault_id` names when you can write to it. A vault you can only read demotes
the capture to your personal memory. Capture never writes
organization-wide memory, whatever your role.
- **What it drops.** Content carrying an email address, a phone number, or a
card number is dropped before any write.
- **What it reports.** Every capture appears in `cruxy.memory_capture`, with the
scope and key of each memory stored and never the value. No field means
nothing was stored.
- **What it never does.** Capture cannot fail or delay a turn. If extraction is
unavailable, it stores nothing.
On a stream, `memory_recall` rides the role primer and `memory_capture` rides
the terminal usage frame, because what a turn stored is known only once the
answer is complete.
## Turn memory off for one request
Any key may send `false` for either side, and `false` needs no scope:
```json
{ "memory": { "recall": false, "capture": false } }
```
`recall: false` reads no memory and ignores `mentioned_vault_ids`.
`capture: false` stores nothing. The two are independent, so
`{ "capture": false }` alone keeps the turn personalized by what is already
stored while adding nothing new.
Pin `Cruxy-Version: 2026-09-26` on every request that relies on `false`. A
gateway that predates the field would ignore it and capture anyway; with the
pin, such a gateway refuses the request with `400` `unknown_version` instead.
A server should use a developer API key. If a signed-in credential does serve
other people, every request must send both `false` values, and `rules: false`,
or the owner's memory and rules reach their turns.
## Review and remove what is stored
- `GET /api/v1/memories` lists your effective memory, one entry per key, with
the `id` of the row that won.
- `DELETE /api/v1/memories/{id}` removes that entry. You can delete what you
can write: your own memories, a vault's where your membership grants
`can_write`, and organization memory if you are an owner or admin.
- `PUT /api/v1/memories` stores a value by hand. A key or value carrying an
email address, a phone number, or a card number is refused with `400`
`pii_rejected`, and nothing is stored.
[API Reference](/docs/en/reference/api) has every field of the memories
endpoints.
## Next steps
- [Standing rules](/docs/en/build/rules): the other per-request context, with the same default split.
- [Get your API key](/docs/en/get-started/api-keys): the two kinds of key and their restrictions.
- [Streaming](/docs/en/build/streaming): where the `cruxy` signals ride on a stream.
- [Errors and retries](/docs/en/build/errors-and-retries): handle `403` `key_scope_forbidden`.
================================================================================
Page: Standing rules
URL: https://learn.getcruxy.com/docs/en/build/rules
Markdown: https://learn.getcruxy.com/docs/en/build/rules.md
================================================================================
# Standing rules
Inject your organization's standing rules into chat as defaults. Rules are on for signed-in keys and off for developer keys unless you opt in.
Standing rules are instructions your organization sets for chat, such as
"Always answer in British English." A vault can have its own. When rules
apply, the gateway injects them into the turn as defaults that the user's own
message overrides, never as constraints that outrank the user. The `rules`
field on a chat request turns injection on or off for that request.
## Defaults depend on the key
| Key | Rules when `rules` is omitted |
| --- | --- |
| Signed-in credential: device login, OAuth login, and the Cruxy web app, desktop, CLI, and mobile | Injected |
| Developer API key or headless key | Not injected, unless the key was issued with allowed scopes that name the `rules` family |
An unscoped developer key has every family but has not opted in, so it gets no
rules by default. A key's rules are its owner's organization's guidance, and a
developer key often serves people other than its owner.
## Opt in
A developer key opts in one of two ways:
- **Per request:** send `rules: true`.
- **Once, on the key:** issue the key with allowed scopes that name the `rules`
family. Every request from it then gets rules unless it sends `rules: false`.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vaani",
"stream": false,
"rules": true,
"messages": [
{ "role": "user", "content": "Draft a two-line status update for the launch." }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'vaani',
stream: false,
rules: true,
messages: [{ role: 'user', content: 'Draft a two-line status update for the launch.' }],
}),
})
const completion = await res.json()
console.log(completion.message.content)
console.log(completion.cruxy.rules_applied) // { count, sources }, or undefined
```
`rules: true` needs a key whose allowed scopes include `rules`. A key without
it gets `403` `key_scope_forbidden` with `required_scope: "rules"`, refused
before anything is billed, and such a key never gets rules. `rules: false`
needs no scope: any key may send it to decline rules for one turn, since the
user's own message already overrides them.
> **Warning:** A model can repeat an injected rule. Anyone chatting through a key that gets
> rules can ask the model to repeat them. Before you opt in a key
> that serves other people, read `GET /api/v1/rules/resolved` and confirm
> nothing in the result is meant to stay internal.
## What gets injected
The switch covers the whole resolved ruleset: a turn gets exactly what
`GET /api/v1/rules/resolved` returns for its context, or nothing.
- **No `vault_id`:** the organization's rules.
- **A `vault_id` for a vault that inherits** (the default): the organization's
rules first, then the vault's own.
- **A `vault_id` for a vault that does not inherit:** the vault's own rules
only.
A `vault_id` you cannot read is treated as none, so the turn runs with
organization rules. Injected rule text is real input and counts in
`usage.input_tokens`. Rules are separate from memory: the `memory` field never
affects them. `POST /v1/chat/completions` never injects rules.
Preview the rules a turn would get. Pass `vault_id` as a query parameter for a
chat inside a vault:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/rules/resolved \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/rules/resolved', {
headers: { Authorization: `Bearer ${process.env.CRUXY_API_KEY}` },
})
const resolved = await res.json()
for (const rule of resolved.rules) {
console.log(rule.source, rule.text) // "org" or "vault", then the instruction
}
```
Each rule carries its `source`, `org` or `vault`, so an unexpected instruction
traces back to the ruleset that contributed it. An empty `rules` array is a
real answer: there is no default ruleset. Unlike chat, this endpoint answers
`404` `vault_not_found` for a `vault_id` you cannot read, so a mistyped id is
reported rather than resolved for the wrong context.
## Know when rules applied
When at least one rule applied, the response carries `cruxy.rules_applied`:
how many rules applied and from which rulesets, never the rule text. The field
is absent when nothing was injected. On a stream it rides the role primer.
```json
{
"cruxy": {
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"routed_tier": "vaani",
"reasoning_effort": "low",
"rules_applied": {
"count": 3,
"sources": [{ "scope": "org", "count": 3 }]
}
}
}
```
## Edit the rules
`GET /api/v1/rules` returns one stored ruleset, the organization's or, with
`vault_id`, a vault's, and `PUT /api/v1/rules` replaces it whole. Its
`permissions.can_write` tells you whether you may change it: the
organization's ruleset is owner or admin only, and a vault's also admits a
member the vault grants `can_write`. A refused write is `403`
`rules_forbidden`.
Rules are re-injected on every request in scope, so each ruleset is capped:
| Cap | Limit |
| --- | --- |
| Rules per ruleset | 50 |
| Characters per rule | 1000 |
| Characters in total | 8000 |
Exceeding a cap is `400` `rules_too_large`. A rule carrying an email address,
a phone number, or a card number is refused with `400` `pii_rejected`, and
nothing is stored. [API Reference](/docs/en/reference/api) has every field of
the rules endpoints.
## Next steps
- [Memory](/docs/en/build/memory): recall and capture, with the same default split.
- [Get your API key](/docs/en/get-started/api-keys#what-a-key-is-allowed-to-do): allowed scopes and other key restrictions.
- [Chat](/docs/en/build/chat): the rest of the request and the `cruxy` block.
================================================================================
Page: Define tools
URL: https://learn.getcruxy.com/docs/en/build/tools
Markdown: https://learn.getcruxy.com/docs/en/build/tools.md
================================================================================
# Define tools
Describe your functions in the tools array with a JSON Schema for their arguments, and use tool_choice to control when the model calls one.
Tool calling lets a chat model ask your code to run a function. You describe
the functions on the request; when the model decides to use one, the turn ends
with the call and its arguments, you run it, and you send the result back. This
page covers the first part. [Handle tool calls](/docs/en/build/tool-calls)
covers answering the calls, and
[Tutorial: build a tool-using agent](/docs/en/build/tool-agent) runs the whole
loop.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"messages": [
{ "role": "user", "content": "What is the weather in Paris?" }
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": { "type": "string", "description": "City name, for example Paris" }
},
"required": ["city"]
}
}
}
],
"tool_choice": "auto"
}'
```
```ts title="TypeScript"
const tools = [
{
type: 'function',
function: {
name: 'get_weather',
description: 'Get the current weather for a city',
parameters: {
type: 'object',
properties: {
city: { type: 'string', description: 'City name, for example Paris' },
},
required: ['city'],
},
},
},
]
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'kavi',
stream: false,
messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
tools,
tool_choice: 'auto',
}),
})
const completion = await res.json()
console.log(completion.finish_reason) // "tool_calls" when the model wants get_weather
console.log(completion.message.tool_calls)
```
## Describe a tool
`tools` is an array. Each entry describes one function:
| Field | Required | What it is |
| --- | --- | --- |
| `type` | Yes | `"function"`. |
| `function.name` | Yes | The name the model calls the function by, and the name it returns in a tool call. |
| `function.description` | No | What the function does, rendered into the prompt the model reads. |
| `function.parameters` | No | A JSON Schema object describing the function's arguments, also rendered into the prompt. |
Write the `description` and each parameter's `description` for the model: say
what the function returns and when to use it, the way you would brief a
colleague.
## Control tool use
`tool_choice` controls whether the model may call your tools, and which:
| `tool_choice` | Effect |
| --- | --- |
| Omitted | The provider default. |
| `"none"` | The model calls no tool. |
| `"auto"` | The model decides whether to call a tool. |
| `"required"` | The model calls at least one tool. |
| `{ "type": "function", "function": { "name": "get_weather" } }` | The model calls that function. This is the only form that makes a tool call deterministic. |
A forced function must appear in `tools`, and `"required"` needs at least one
tool. Either mistake is `400` `invalid_request`, answered before anything goes
upstream.
## Schema bounds
The gateway bounds each tool's `parameters` schema by size, depth, and node
count. A schema past a bound is `400` `invalid_tool_schema`, which adds three
fields:
| Field | Meaning |
| --- | --- |
| `tool` | The function whose schema broke the bound. |
| `bound` | Which bound: `bytes`, `depth`, or `nodes`. |
| `limit` | That bound's value. |
Retrying never helps, because the fix belongs in the tool definitions your
client ships. To check them ahead of time, send the same `tools` to
`POST /api/v1/tokens/count`: it applies the same bounds with the same `400`
answer, is not billed, and returns the input tokens the request would bill,
tools included.
## Tools are billed input
The provider renders each function's name, description, and parameters into
the prompt, so every request that sends `tools` is billed for them as input
tokens. Count them with `POST /api/v1/tokens/count`, and keep descriptions as
long as the model needs and no longer.
## Where tools work
`tools` and `tool_choice` are accepted on `POST /api/v1/chat`,
`POST /api/v2/chat`, and `POST /v1/chat/completions`. Each tier reports
`supports_tools` in `GET /api/v1/models`; [Models](/docs/en/reference/models)
shows how to read the catalog.
## Next steps
- [Handle tool calls](/docs/en/build/tool-calls): run the calls the model returns and send the results back.
- [Tutorial: build a tool-using agent](/docs/en/build/tool-agent): the full loop in one program.
- [Chat](/docs/en/build/chat): the rest of the request and the response.
================================================================================
Page: Handle tool calls
URL: https://learn.getcruxy.com/docs/en/build/tool-calls
Markdown: https://learn.getcruxy.com/docs/en/build/tool-calls.md
================================================================================
# Handle tool calls
Read the tool calls the model returns, run each function, send every result back by tool_call_id, and repeat until finish_reason is stop.
When the model decides to call one of the tools you defined, the turn ends with
`finish_reason: "tool_calls"` and the assistant message carries `tool_calls`.
You run each function, then send a new request with the results; the model
reads them and continues. [Define tools](/docs/en/build/tools) covers the
request that offers the tools.
## Read the tool calls
On `POST /api/v1/chat` with `stream: false`, the assistant reply is the
top-level `message`. A turn that calls a tool looks like this, trimmed to the
fields that matter here:
```json
{
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
}
]
},
"finish_reason": "tool_calls"
}
```
Each entry in `tool_calls` has three fields:
| Field | What it is |
| --- | --- |
| `id` | The call's id. Your result for this call carries it back as `tool_call_id`. |
| `type` | `"function"`. |
| `function` | The `name` of the function to run, and its `arguments`. |
`arguments` is a JSON-encoded string, not an object. Parse it before you
dispatch:
```ts
const args = JSON.parse(call.function.arguments) // { city: "Paris" }
```
## Send the results back
1. **Append the assistant message whole**
Push the `message` the gateway returned onto your conversation, `tool_calls`
included. The model needs its own call in the history next to the result that
answers it.
2. **Append one tool message per call**
For every entry in `tool_calls`, run the function and append a message with
`role: "tool"`, the call's `id` as `tool_call_id`, and the result as a string in
`content`.
3. **Call chat again with the same tools**
Send the whole conversation back with the same `tools`. The model reads the
results and either answers or calls another tool.
This request carries a finished tool exchange back to the model:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"messages": [
{ "role": "user", "content": "What is the weather in Paris?" },
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
}
]
},
{
"role": "tool",
"tool_call_id": "call_abc123",
"content": "{\"city\":\"Paris\",\"tempC\":18,\"sky\":\"clear\"}"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
}
}
}
]
}'
```
```ts title="TypeScript"
const tools = [
{
type: 'function',
function: {
name: 'get_weather',
description: 'Get the current weather for a city',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
},
]
// The assistant turn the gateway returned, kept whole.
const assistantTurn = {
role: 'assistant',
content: '',
tool_calls: [
{
id: 'call_abc123',
type: 'function',
function: { name: 'get_weather', arguments: '{"city":"Paris"}' },
},
],
}
const messages = [
{ role: 'user', content: 'What is the weather in Paris?' },
assistantTurn,
// Your result, tied back to the call by its id.
{
role: 'tool',
tool_call_id: 'call_abc123',
content: JSON.stringify({ city: 'Paris', tempC: 18, sky: 'clear' }),
},
]
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ model: 'kavi', stream: false, messages, tools }),
})
const completion = await res.json()
console.log(completion.finish_reason) // "stop" once the model has its answer
console.log(completion.message.content)
```
## Loop on finish_reason
`finish_reason` tells you what to do with each turn:
| `finish_reason` | What it means | What you do |
| --- | --- | --- |
| `tool_calls` | The model wants the calls in `message.tool_calls`. | Run them, append the results, and call again. |
| `stop` | The model finished its reply. | Read the answer in `message.content`. |
| `length` | The output hit `max_tokens` or the tier's cap. | Raise `max_tokens` or lower `reasoning_effort`; see [Reasoning effort](/docs/en/build/reasoning-effort). |
| `content_filter` | The upstream model stopped for safety. | Stop the loop and handle the turn as unanswered. |
Keep calling while `finish_reason` is `tool_calls`, and stop when it returns
`stop`. Put a ceiling on the number of rounds too, so a model that keeps
calling tools cannot run your loop forever.
## Several calls in one turn
`tool_calls` is an array, so one turn can ask for several calls. Run all of
them and append one `tool` message for each before you call chat again. The
`tool_call_id` on each result is what pairs it with its call.
If a function fails, still answer its call: put a short description of the
failure in that tool message's `content`, so every call in the turn has a
result.
## Send the tools on every request
Keep sending the same `tools` for as long as the conversation holds tool
exchanges. When the history replays a tool call to a function that the
request's `tools` does not declare, the gateway sends that exchange to the
model as prose instead of as a tool call, and reports it in
`cruxy.undeclared_tool_replay`:
| Field | Meaning |
| --- | --- |
| `exchanges` | How many assistant messages were sent as prose. |
| `tool_calls` | How many tool calls those messages carried. |
The field is absent when nothing was rewritten, and nothing you store or send
is changed: only what the model reads that turn.
## Tool calls on a stream
With `stream: true`, the calls arrive in pieces. Deltas carry `tool_calls`
fragments instead of `content`; each fragment has an `index` and a piece of
`function.arguments`, and may carry the call's `id`, `type`, and
`function.name`. Assemble the fragments per `index`, concatenating the
`arguments` pieces. The finish frame with `finish_reason: "tool_calls"` means
the calls are complete; answer them as above.
[Tool-call deltas](/docs/en/build/streaming#tool-call-deltas) shows the frames.
On the native wire, `POST /api/v2/chat`, the fragments ride `delta` events and
the terminal `finish` event carries `finish_reason`.
## Next steps
- [Tutorial: build a tool-using agent](/docs/en/build/tool-agent): this loop as one working program.
- [Define tools](/docs/en/build/tools): the tool shape, `tool_choice`, and schema bounds.
- [Streaming](/docs/en/build/streaming): the full frame order.
================================================================================
Page: Tutorial: build a tool-using agent
URL: https://learn.getcruxy.com/docs/en/build/tool-agent
Markdown: https://learn.getcruxy.com/docs/en/build/tool-agent.md
================================================================================
# Tutorial: build a tool-using agent
Build a working TypeScript agent that offers two tools, runs the calls the model returns, and loops on finish_reason until it answers.
In this tutorial you build a small agent: one TypeScript file that answers a
travel question by calling two functions of yours. It uses `POST /api/v1/chat`
with `stream: false` and the loop from
[Handle tool calls](/docs/en/build/tool-calls). You finish with a program you
can run, and a loop you can reuse with your own tools.
## Before you start
- Node.js 22 or later, for the built-in `fetch`.
- A developer API key in `CRUXY_API_KEY`; see
[Get your API key](/docs/en/get-started/api-keys).
- A way to run TypeScript directly. The commands here use `npx tsx`.
## Step 1: Watch one tool call
Before writing the loop, send one request by hand and look at what comes back.
Offer a `get_weather` tool and ask about the weather:
```bash title="curl"
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kavi",
"stream": false,
"messages": [
{ "role": "user", "content": "What is the weather in Paris?" }
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
}
}
}
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'kavi',
stream: false,
messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
tools: [
{
type: 'function',
function: {
name: 'get_weather',
description: 'Get the current weather for a city',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
},
],
}),
})
const { message, finish_reason } = await res.json()
console.log(finish_reason) // "tool_calls"
console.log(message.tool_calls)
```
When the model chooses the tool, `finish_reason` is `"tool_calls"` and
`message.tool_calls` holds the call: an `id`, the function `name`, and its
`arguments` as a JSON string. The model does not run anything. Your program
runs the function and sends the result back, and the rest of this tutorial
automates that.
## Step 2: Describe the tools
Create `agent.mts`. Start with the types the loop passes around and the two
tools the model may call:
```ts
const BASE = 'https://api.cruxy.ai'
interface ToolCall {
id: string
type: 'function'
function: { name: string; arguments: string }
}
type Message =
| { role: 'system' | 'user'; content: string }
| { role: 'assistant'; content: string; tool_calls?: ToolCall[] }
| { role: 'tool'; tool_call_id: string; content: string }
interface Completion {
message: { role: 'assistant'; content: string; tool_calls?: ToolCall[] }
finish_reason: 'stop' | 'tool_calls' | 'length' | 'content_filter'
}
const tools = [
{
type: 'function',
function: {
name: 'get_weather',
description: 'Get the current weather for a city: temperature in Celsius and the sky.',
parameters: {
type: 'object',
properties: { city: { type: 'string', description: 'City name, for example Paris' } },
required: ['city'],
},
},
},
{
type: 'function',
function: {
name: 'get_local_time',
description: 'Get the current local time in a city.',
parameters: {
type: 'object',
properties: { city: { type: 'string', description: 'City name, for example Mumbai' } },
required: ['city'],
},
},
},
]
```
[Define tools](/docs/en/build/tools) covers every field of a tool.
## Step 3: Write the handlers
Each tool name maps to a function of yours. `runTool` parses the call's
`arguments`, runs the matching handler, and always returns a string for the
tool message, even when the call names an unknown tool or the handler throws:
```ts
const TIME_ZONES: Record = {
paris: 'Europe/Paris',
mumbai: 'Asia/Kolkata',
london: 'Europe/London',
}
const handlers: Record Promise> = {
async get_weather({ city }) {
// Replace with a call to your weather service.
return { city, tempC: 18, sky: 'clear' }
},
async get_local_time({ city }) {
const timeZone = TIME_ZONES[city.toLowerCase()]
if (!timeZone) return { city, error: 'unknown city' }
return { city, time: new Date().toLocaleTimeString('en-GB', { timeZone }) }
},
}
async function runTool(call: ToolCall): Promise {
const handler = handlers[call.function.name]
if (!handler) return JSON.stringify({ error: `no tool named ${call.function.name}` })
try {
const result = await handler(JSON.parse(call.function.arguments))
return JSON.stringify(result)
} catch (err) {
return JSON.stringify({ error: String(err) })
}
}
```
## Step 4: Call the model
`chat` sends the conversation so far, with the tools, and returns the turn. It
throws on an error response, naming the stable `code` and the `request_id`:
```ts
async function chat(messages: Message[]): Promise {
const res = await fetch(`${BASE}/api/v1/chat`, {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ model: 'kavi', stream: false, messages, tools }),
})
if (!res.ok) {
const { code, error, request_id } = await res.json()
throw new Error(`${res.status} ${code}: ${error} (request ${request_id})`)
}
return res.json()
}
```
The request leaves `max_tokens` unset. `kavi` thinks by default, and its
thinking shares the `max_tokens` budget, so a small cap could end a turn with
`finish_reason: "length"` before the model writes a call or an answer; see
[Reasoning effort](/docs/en/build/reasoning-effort).
## Step 5: Run the loop
The loop calls the model, keeps the assistant turn, and branches on
`finish_reason`. On `tool_calls` it answers every call by `id` and goes around
again; on `stop` the turn's `content` is the answer:
```ts
async function runAgent(question: string, maxRounds = 8): Promise {
const messages: Message[] = [{ role: 'user', content: question }]
for (let round = 0; round < maxRounds; round++) {
const { message, finish_reason } = await chat(messages)
messages.push(message) // the assistant turn, tool_calls and all
if (finish_reason === 'tool_calls') {
for (const call of message.tool_calls ?? []) {
messages.push({ role: 'tool', tool_call_id: call.id, content: await runTool(call) })
}
continue
}
if (finish_reason === 'stop') return message.content
throw new Error(`the turn ended with finish_reason "${finish_reason}"`)
}
throw new Error(`no answer after ${maxRounds} rounds`)
}
console.log(
await runAgent(
'I fly from Mumbai to Paris tonight. What is the weather in Paris, and what time is it there now?',
),
)
```
## Step 6: Run it
With the four pieces in `agent.mts`, in order, run:
```bash
npx tsx agent.mts
```
The program prints the model's answer. Along the way the model asks for
`get_weather` and `get_local_time`, possibly both in one turn, and the loop
answers each call before it asks again.
## Why the loop is shaped this way
- **It branches on `finish_reason`.** `tool_calls` means the model is waiting
on your results, and `stop` means it has answered. `length` and
`content_filter` end the loop with an error, because neither carries a
finished answer.
- **It appends the assistant turn whole.** The model needs to see its own call
next to the result that answers it, so the turn goes into the history,
`tool_calls` included, before any result.
- **It answers every call by `id`.** One turn can request several calls. Each
result carries its call's `id` as `tool_call_id`, which is how the model
pairs results with calls.
- **It sends the same `tools` every round.** A replayed call to a tool the
request does not declare reaches the model as prose, not as a tool call.
- **It always returns a string from `runTool`.** A failed or unknown call still
gets a result, so no call goes unanswered.
- **It stops after `maxRounds`.** The bound is a backstop for a model that
never settles; reaching it is worth logging, not presenting as an answer.
## Next steps
- [Handle tool calls](/docs/en/build/tool-calls): the response shapes the loop reads, and tool calls on a stream.
- [Define tools](/docs/en/build/tools): `tool_choice` and the schema bounds.
- [Errors and retries](/docs/en/build/errors-and-retries): decide which errors `chat` should retry.
================================================================================
Page: Reliability
URL: https://learn.getcruxy.com/docs/en/build/reliability
Markdown: https://learn.getcruxy.com/docs/en/build/reliability.md
================================================================================
# Reliability
Run Cruxy in production: per-tier deadlines, which errors to retry and how, Idempotency-Key, telling a finished stream from a failed one, and status.
This page collects what the gateway guarantees when a call is slow or fails,
and what it leaves to your client. Each section links to the page that covers
its topic in full. The sample streams a long `kavi` turn, accepts the answer
only once the stream finishes, and retries a failed attempt:
```bash title="curl"
# One key per logical request: reuse it if you send this request again.
IDEMPOTENCY_KEY=$(uuidgen)
# A long kavi turn: stream it. --max-time sits above the five-minute limit
# on a stream, so the gateway, not curl, decides when the turn has run long.
curl -sN --max-time 330 https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
-d '{
"model": "kavi",
"stream": true,
"messages": [
{ "role": "user", "content": "Plan a three-day walking tour of Old Delhi." }
]
}' > stream.txt
# The completion finished only if the stream ended with [DONE].
grep -q '^data: \[DONE\]' stream.txt && echo "complete" || echo "incomplete: send it again"
```
```ts title="TypeScript"
// Your retry budget and timeout. Tune them for your workload; they are not API
// limits. A stream may run for up to five minutes, so the timeout sits above it.
const MAX_ATTEMPTS = 3
const BASE_DELAY_MS = 500
const TIMEOUT_MS = 330_000
// Transient failures the API marks retryable.
const RETRYABLE = new Set([
'rate_limited',
'credits_unavailable',
'headless_concurrency_limit',
'limits_unavailable',
'upstream_overloaded',
'model_unavailable',
'all_models_unavailable',
'server_shutting_down',
'internal_error',
])
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
// Exponential backoff with full jitter.
const backoff = (attempt: number) => Math.random() * BASE_DELAY_MS * 2 ** attempt
type Attempt = { answer: string } | { retryInMs: number }
async function streamOnce(
body: Record,
idempotencyKey: string,
attempt: number,
): Promise {
let res: Response
try {
res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey,
},
body: JSON.stringify({ ...body, stream: true }),
signal: AbortSignal.timeout(TIMEOUT_MS),
})
} catch {
return { retryInMs: backoff(attempt) } // no response at all: send it again
}
if (!res.ok) {
const { code, error, request_id } = await res.json()
if (!RETRYABLE.has(code)) throw new Error(`${res.status} ${code}: ${error} (request_id ${request_id})`)
const retryAfter = Number(res.headers.get('Retry-After'))
return { retryInMs: retryAfter > 0 ? retryAfter * 1000 : backoff(attempt) }
}
if (!res.body) return { retryInMs: backoff(attempt) }
const reader = res.body.getReader()
const decoder = new TextDecoder()
let buffer = ''
let answer = ''
try {
while (true) {
const { value, done } = await reader.read()
if (done) break
buffer += decoder.decode(value, { stream: true })
const lines = buffer.split('\n')
buffer = lines.pop() ?? '' // keep a partial line for the next read
for (const raw of lines) {
const line = raw.trim()
if (!line.startsWith('data:')) continue
const data = line.slice('data:'.length).trim()
if (data === '[DONE]') return { answer } // the only success signal
const choice = JSON.parse(data).choices[0] // undefined on the usage frame
if (choice?.delta.content) answer += choice.delta.content
}
}
} catch {
// The connection dropped mid-stream: fall through and treat it as a failure.
}
// The stream ended before [DONE]: a failed completion. Discard the partial answer.
return { retryInMs: backoff(attempt) }
}
async function chat(body: Record): Promise {
// One key per logical request, reused on every retry of it: if two attempts
// both complete, the request is billed once.
const idempotencyKey = crypto.randomUUID()
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
const result = await streamOnce(body, idempotencyKey, attempt)
if ('answer' in result) return result.answer
if (attempt < MAX_ATTEMPTS) await sleep(result.retryInMs)
}
throw new Error(`no complete answer after ${MAX_ATTEMPTS} attempts`)
}
const answer = await chat({
model: 'kavi',
messages: [{ role: 'user', content: 'Plan a three-day walking tour of Old Delhi.' }],
})
console.log(answer)
```
## Deadlines and timeouts
With `stream: false`, the gateway makes one attempt at the model, bounded by the
tier's deadline:
| Tier | Deadline |
| --- | --- |
| `kavi` | 90 seconds |
| `vaani` | 90 seconds |
| `mira` | 45 seconds |
An attempt that runs past its deadline is cancelled and answered `504`
`upstream_timeout`. Nothing is billed, and the gateway does not retry it, since
a turn that needed longer needs longer again. The same deadlines apply on all
three chat wires: `POST /api/v1/chat`, `POST /api/v2/chat`, and
`POST /v1/chat/completions`.
- **Stream long turns.** A stream may run for up to five minutes. A long `kavi`
turn, with reasoning on and a large output, belongs on a stream.
- **Or ask for less.** A lower `max_tokens` or `reasoning_effort` shortens the
turn; see [Reasoning effort](/docs/en/build/reasoning-effort).
- **Set your client timeout above the deadline,** or above five minutes for a
stream, so a slow model reaches you as a `504` with a `request_id` rather
than as your own abort.
## What the gateway retries for you
- **A failing model, inside the gateway.** An overloaded or failing model, or a
dropped connection to it, is retried a bounded number of times, with
backoff, before the gateway answers `503` `upstream_overloaded` or
`model_unavailable`. On a stream, that happens only while the stream opens,
never after a byte has reached you.
- **Never a non-streaming timeout.** See [Deadlines and timeouts](#deadlines-and-timeouts).
- **Never on another tier.** Once a call is sent, it stays on its tier.
`model: "auto"` picks a healthy tier before the call, and an explicit tier
that keeps failing is refused at once with `503` `model_unavailable` while it
cools down. See [Auto routing](/docs/en/build/auto-routing).
## What to retry
Branch on the error's `code`, never on its message:
| Response | What to do |
| --- | --- |
| `429` `rate_limited`, `credits_unavailable`, `headless_concurrency_limit`, or `503` `limits_unavailable` | Wait the `Retry-After` seconds, then retry. |
| `503` `upstream_overloaded`, `model_unavailable`, `all_models_unavailable`, `server_shutting_down`, or `500` `internal_error` | Back off, then retry. |
| `504` `upstream_timeout` | Change the request first: stream it, or lower `max_tokens` or `reasoning_effort`. |
| `429` `budget_exhausted`, `key_spend_cap_exceeded`, `workspace_spend_cap_exceeded`, `credits_exhausted` | Stop. Each clears only at its `reset_at`, or when the cap changes. |
| No response, or a stream that ended before it finished | Send the request again, with the same `Idempotency-Key`. |
| Any other code | Look it up in [Errors and retries](/docs/en/build/errors-and-retries#which-failures-to-retry). Most describe the request or the key, and fail the same way until you change them. |
When you retry:
- **Honour `Retry-After`.** Every `429` sends it, and so does a `503`
`limits_unavailable`.
- **Otherwise back off exponentially, with jitter,** so many clients that failed
together do not retry together.
- **Cap the attempts and the total wait,** so a persistent failure ends instead
of hanging.
- **Log the `request_id` of every failure,** retried or not; see
[Request IDs and metadata](/docs/en/build/request-ids).
## Retry safely with Idempotency-Key
Send an `Idempotency-Key` header, one per logical request, and reuse it on every
retry of that request:
- **A success is stored for 24 hours.** A later or concurrent call with the same
key and the same body replays it byte for byte, without calling the model and
without billing again. A concurrent duplicate waits for the first call to
finish.
- **Only a success is stored.** An error, a `429` or any `5xx` included,
releases the key, so a retry with the same key and body runs the request
again. That is the intended way to retry.
- **A different body under the same key** is `409` `idempotency_conflict`.
- **A response over 256 KiB** is served but not stored, so a retry runs again.
It is still billed once.
- **A key is at most 255 characters,** and is scoped to your API key and the
endpoint.
- **A stream stores and replays nothing.** A retry with the same key and body
streams a new generation. When both attempts complete, the request is billed
once; an attempt that did not complete is billed for the tokens it delivered,
and the retry is billed as well.
[Errors and retries](/docs/en/build/errors-and-retries#make-retries-idempotent)
lists the endpoints that accept the header.
## Confirm a stream finished
Once a stream opens, its `200` is already sent, so a failure after that point
cannot change the status. How the stream tells you it failed depends on the
wire:
- **`POST /api/v1/chat`:** `data: [DONE]` is the only success signal. A failure
after the stream opens sends no error frame; the stream stops early. Treat a
stream that ends before `[DONE]` as a failed completion: discard the partial
output or retry, and never show it as a finished answer.
- **`POST /api/v2/chat`:** the stream always ends with exactly one `finish` or
`error` frame. It has no `[DONE]`.
- **`POST /v1/chat/completions`:** a failure after the stream opens ends it with
a `data: {"error": {...}}` frame instead of `[DONE]`.
The tokens a failed stream delivered are billed as a partial usage row.
[Streaming](/docs/en/build/streaming#done-is-the-only-success-signal) shows how
to detect each case.
## Check platform status
[status.getcruxy.com](https://status.getcruxy.com) is the Cruxy status page. For
the same picture in code, `GET /api/status` returns a status document. It takes
no key.
```bash title="curl"
curl https://api.cruxy.ai/api/status
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/status')
// A 503 still carries the document, with status "unknown": no model component
// has a fresh measurement, so the document cannot be trusted.
const status = await res.json()
console.log(`platform: ${status.status}`)
// Iterate the array and read each row's kind; new components are appended.
for (const component of status.components) {
console.log(`${component.id} (${component.kind}): ${component.status}`)
}
```
- **`status`** is the whole platform in one word: `healthy`, `degraded`,
`down`, or `unknown`.
- **`components`** has a row for the API, the database, each chat tier, and
the `cruxy-embed`, `cruxy-rerank`, and `cruxy-guard` models, each with its own
`status` and when it was last checked. A component that is measurably down
is a `200` carrying `down`; the document is a `503` only when no model
component has a fresh measurement.
- **The verdicts come from active probes,** not from live traffic, and the
response is never cached.
`GET /api/health` reports only that the gateway's HTTP listener is up. It checks
no model or database, so do not read it as platform status.
## Production checklist
- **Use a developer API key on a server.** See
[Get your API key](/docs/en/get-started/api-keys).
- **Pin `Cruxy-Version`,** and ignore JSON fields you do not recognize.
- **Set client timeouts above the tier deadline,** or above five minutes for a
stream.
- **Stream long `kavi` turns,** and size `max_tokens` for the thinking as well
as the answer.
- **Send an `Idempotency-Key`** on every request that takes one, and reuse it
on that request's retries.
- **Retry only what the table above says,** honouring `Retry-After`, with a
cap on attempts.
- **Check for `[DONE]`,** or a `finish` frame on v2, before you show a streamed
answer.
- **Log the `X-Request-Id`, the `request_id`, and the `Cruxy-Version`** of
every call.
- **Read [Known limitations](/docs/en/build/known-limitations)** before you
launch.
## Next steps
- [Errors and retries](/docs/en/build/errors-and-retries): every error code, and a retrying client.
- [Streaming](/docs/en/build/streaming): the frames, and a complete stream reader.
- [Rate limits and budgets](/docs/en/build/rate-limits): read your headroom before you hit a `429`.
- [Known limitations](/docs/en/build/known-limitations): what the API does not do yet, and the workarounds.
================================================================================
Page: Errors and retries
URL: https://learn.getcruxy.com/docs/en/build/errors-and-retries
Markdown: https://learn.getcruxy.com/docs/en/build/errors-and-retries.md
================================================================================
# Errors and retries
Read the error envelope, look up every error code the API defines, and decide which failures to retry and how long to wait.
A failed request returns a non-2xx status and a JSON error body. Read the HTTP
status first, then branch on the body's `code`: the code is stable, while the
wording of the message can change.
## The error envelope
```json
{
"code": "invalid_request",
"error": "invalid JSON body",
"request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
```
| Field | Meaning |
| --- | --- |
| `code` | A stable, machine-readable error code. Branch on this. |
| `error` | A human-readable message. Log it; do not parse it. |
| `request_id` | The id of the failed call, equal to the response's `X-Request-Id` header. Log it with every failure; see [Request IDs and metadata](/docs/en/build/request-ids). |
Some codes add fields next to these three. Each is optional, and present only
on the codes listed:
| Field | Sent on |
| --- | --- |
| `tier` | The tier a model failure or context check was about: `upstream_error`, `upstream_overloaded`, `upstream_timeout`, `model_unavailable`, and `server_shutting_down` on chat, and `context_overflow`. |
| `reason` | Why: `rate` or `concurrency` on `rate_limited` from chat; why a document's text could not be read on `file_not_text`. |
| `retry_after` | The `Retry-After` value in seconds: `rate_limited`, `limits_unavailable`, `credits_unavailable`, `headless_concurrency_limit`. |
| `reset_at` | When the refusal clears (RFC 3339): `budget_exhausted`, `key_spend_cap_exceeded`, `workspace_spend_cap_exceeded`, `credits_exhausted`. `null` on `model_not_allowed`, since waiting never clears an allowlist. |
| `model`, `state`, `window`, `mira_available` | A spend or budget refusal: `budget_exhausted`, `key_spend_cap_exceeded`, `workspace_spend_cap_exceeded`, `model_not_allowed`. `window` is `monthly` or `burst`; `mira_available` says whether the same request on `mira` would be served now. |
| `balance`, `currency` | A headless key's remaining credit, on `credits_exhausted`. |
| `dimension` | The check that faulted on `limits_unavailable`: `rate` or `budget`. |
| `required_scope`, `allowed_scopes` | The family the key would need and the families it has, on `key_scope_forbidden`. |
| `required_version` | The `Cruxy-Version` a request field needs pinned, on `invalid_request`. |
| `hazard_codes` | The blocking classifier codes behind `moderation_rejected`. |
| `tool`, `bound`, `limit` | The function and the cap it broke, on `invalid_tool_schema`. `bound` and `limit` also name the cap on `file_too_large`, and `limit` is the context window on `context_overflow`. |
| `file_id`, `filename`, `content_type`, `tokens` | The refused file, on `file_not_found`, `file_not_text`, `file_too_large`, and `image_not_supported`. |
| `vision_tiers` | The tiers that read images, heaviest first, on `image_not_supported`. |
| `measured_bytes`, `limit_bytes` | The size and the cap on `input_not_countable`. |
Ignore fields you do not recognize.
Three surfaces use a different shape:
- **`POST /v1/chat/completions`** uses the OpenAI envelope,
`{"error": {"message", "type", "param", "code"}}`. Its `code` is the same
Cruxy code listed here; `type` is a coarse category derived from the status
(`invalid_request_error`, `authentication_error`, `permission_error`,
`not_found_error`, `rate_limit_error`, or `server_error`); `param` is always
`null`.
- **Device login** answers a pending or refused poll with an OAuth-style body;
see [Get your API key](/docs/en/get-started/api-keys#device-login).
- **A `POST /api/v2/chat` stream** that fails after it opens ends with an
`error` event carrying `code`, `error`, and `request_id`.
## Handle errors and retry safely
This request sends an `Idempotency-Key`, so once it succeeds, running it again
with the same key replays the stored response instead of calling the model and
billing again. The gateway stores only a success, so a retry after an error
runs the request again under the same key. The TypeScript version retries the
codes the API marks retryable, waiting `Retry-After` when it is sent, and
retries a request that got no response with the same key:
```bash title="curl"
IDEMPOTENCY_KEY=$(uuidgen)
# -i prints the status line and headers, including Cruxy-Version and the
# X-RateLimit-* headers. Run it twice: once the first call succeeds, the
# second replays it.
curl -i https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
-d '{
"model": "vaani",
"stream": false,
"messages": [
{ "role": "user", "content": "Hello" }
]
}'
```
```ts title="TypeScript"
class CruxyError extends Error {
constructor(
readonly status: number,
readonly code: string,
readonly requestId: string,
message: string,
) {
super(`${status} ${code}: ${message} (request_id ${requestId})`)
}
}
// Your retry budget and timeout. Tune them for your workload; they are not API limits.
// TIMEOUT_MS sits above the longest tier deadline (90 s), so a slow model
// reaches you as 504 upstream_timeout rather than as your own abort.
const MAX_ATTEMPTS = 4
const BASE_DELAY_MS = 500
const TIMEOUT_MS = 120_000
// Transient failures the API marks retryable. upstream_timeout is left out:
// the gateway already made its one attempt, and the same request usually
// times out again.
const RETRYABLE = new Set([
'rate_limited',
'credits_unavailable',
'headless_concurrency_limit',
'limits_unavailable',
'upstream_overloaded',
'model_unavailable',
'all_models_unavailable',
'server_shutting_down',
'internal_error',
])
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
// Exponential backoff with full jitter.
const backoff = (attempt: number) => Math.random() * BASE_DELAY_MS * 2 ** attempt
async function chat(body: Record) {
// One key per logical request, reused on every retry of it.
const idempotencyKey = crypto.randomUUID()
for (let attempt = 1; ; attempt++) {
let res: Response
try {
res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey,
},
body: JSON.stringify({ ...body, stream: false }),
signal: AbortSignal.timeout(TIMEOUT_MS),
})
} catch (err) {
// No response: the call may or may not have run. If it succeeded, the
// same key replays its stored response rather than running and billing
// it a second time; if it failed, the same key runs it again.
if (attempt >= MAX_ATTEMPTS) throw err
await sleep(backoff(attempt))
continue
}
if (res.ok) return res.json()
const { code, error, request_id } = await res.json()
// An error is never stored, so the same key runs the request again.
if (RETRYABLE.has(code) && attempt < MAX_ATTEMPTS) {
const retryAfter = Number(res.headers.get('Retry-After'))
await sleep(retryAfter > 0 ? retryAfter * 1000 : backoff(attempt))
continue
}
throw new CruxyError(res.status, code, request_id, error)
}
}
const completion = await chat({
model: 'vaani',
messages: [{ role: 'user', content: 'Hello' }],
})
console.log(completion.message.content)
```
## Which failures to retry
| Failure | Retry? | How |
| --- | --- | --- |
| `429` `rate_limited`, `credits_unavailable`, `headless_concurrency_limit` | Yes | Wait `Retry-After` seconds, then retry. |
| `503` `upstream_overloaded`, `model_unavailable`, `all_models_unavailable`, `server_shutting_down`, `limits_unavailable` | Yes | Back off, honouring `Retry-After` where it is sent. The gateway already retried the model before it answered `upstream_overloaded` or `model_unavailable`. |
| `500` `internal_error` | Yes | Back off. A write may have run, so retry with the same `Idempotency-Key` where the route takes one. |
| `504` `upstream_timeout` | Only if changed | The same request usually times out again. Stream it, or lower `max_tokens` or `reasoning_effort`; see [Timeouts](#timeouts). |
| `429` `budget_exhausted`, `key_spend_cap_exceeded`, `workspace_spend_cap_exceeded`, `credits_exhausted` | Not now | The allowance clears only at `reset_at`, or when the cap changes. `Retry-After` counts down to that moment. |
| `422` `structured_output_unsatisfied` | Yes | It may pass on a retry. The attempt was billed; a retry with the same key and body is not billed again. |
| `409` `idempotency_conflict` | Yes, adjusted | Resend the original body to replay the stored response, or send the new body under a new key. |
| No response at all: a dropped connection or your own timeout | Yes | Retry with the same `Idempotency-Key`. |
| A stream that ended before `data: [DONE]` | Yes | Send the request again; see [Streaming](/docs/en/build/streaming#done-is-the-only-success-signal). |
| `502` `upstream_error` from a model call | No | The model provider rejected the request, and it fails the same way again. From native sign-in it is retryable. |
| `503` `count_unavailable` | No | A fact of the deployment; it does not clear. |
| Every other code | No | It describes the request or the credential. Fix the request, the key, or the target first. |
[Rate limits and budgets](/docs/en/build/rate-limits) explains the difference
between a rate limit and a budget, and how to read your remaining headroom.
### Timeouts
With `stream: false`, the gateway makes one attempt at the model, bounded by
the tier's deadline:
| Tier | Deadline |
| --- | --- |
| `kavi` | 90 seconds |
| `vaani` | 90 seconds |
| `mira` | 45 seconds |
An attempt that runs past the deadline is cancelled and answered `504`
`upstream_timeout`, and nothing is billed. The gateway does not retry a
timeout: a turn that needed longer needs longer again. Send `stream: true`
instead, since a stream may run for up to five minutes, or ask for less with a
lower `max_tokens` or `reasoning_effort`. [Reliability](/docs/en/build/reliability)
collects the deadlines, retries, and stream checks in one place.
Other upstream failures, such as an overloaded or failing model or a dropped
connection, are retried inside the gateway a bounded number of times, with
backoff, before it answers `503` `upstream_overloaded` or `model_unavailable`.
On `stream: true` that happens only while the stream opens, never after a byte
has reached you. A call never moves to another tier once it is sent:
`model: "auto"` picks a healthy tier before the call, and an explicit tier that
keeps failing is refused at once with `503` `model_unavailable` while it cools
down.
### Back off
- **Honour `Retry-After`.** Every `429` carries it, and so does a `503`
`limits_unavailable`. On `rate_limited`, `X-RateLimit-Reset` gives the same
moment as an absolute Unix time in seconds. On a budget or spend-cap refusal,
`Retry-After` counts down to `reset_at` instead, while the `X-RateLimit-*`
headers still describe the per-minute request window.
- **Otherwise back off exponentially, with jitter.** Double the wait after each
failed attempt and randomize it, so many clients that failed together do not
retry together.
- **Cap the attempts and the total wait.** A retry budget keeps a persistent
failure from turning into a stuck request.
- **Log `request_id` for every failure,** retried or not.
### Make retries idempotent
`Idempotency-Key` makes a retry safe. The first call with a key is served, and
when it succeeds its response is stored for 24 hours. A later or concurrent
call with the same key and the same body replays that response byte for byte,
without calling the model again and without billing again; a concurrent
duplicate waits for the first call to finish.
Only a success is stored. An error, a `429` or any `5xx` included, releases the
key, so a retry with the same key and body runs the request again. Reusing the
key is the intended way to retry a `429` or a `5xx`.
- **Scope:** a key belongs to your API key and to the endpoint. On a
conversation append, it belongs to the conversation.
- **Conflicts:** the same key with a different body is `409`
`idempotency_conflict`.
- **Large responses:** a response over 256 KiB is served but not stored, so a
retry runs again. It is still billed once.
- **Length:** keys are at most 255 characters.
These endpoints accept it:
- `POST /api/v1/chat`, `POST /api/v2/chat`, and `POST /v1/chat/completions`
- `POST /api/v1/embeddings`, `POST /api/v1/rerank`, and `POST /api/v1/moderations`
- `POST /api/v1/conversations/{id}/messages`, which stores its `201` and
replays it without appending a second message
> **Note:** A `stream: true` chat stores and replays nothing, because a stream cannot be
> replayed. A retry with the same key and body streams a new generation; when
> both attempts complete, the request is billed once. An attempt that did not
> complete is billed for the tokens it delivered, and the retry is billed as well.
## Error codes
Every code the API defines, grouped by the status it is returned with.
### 400 Bad Request
The request itself is wrong. Fix it; sending it again unchanged fails the same
way.
| `code` | Meaning |
| --- | --- |
| `invalid_request` | Malformed JSON, or a field that is missing or invalid. Carries `required_version` when the request uses a field that needs a newer `Cruxy-Version` pin. |
| `context_overflow` | The input exceeds the tier's context window. |
| `invalid_schema` | A `response_format` JSON schema is malformed, oversized, too deep, or uses an unsupported keyword. |
| `invalid_tool_schema` | A tool's `parameters` schema exceeds a bound. Adds `tool`, `bound`, and `limit`. The fix belongs in your client, so a retry cannot help. |
| `file_not_text` | A file named in `file_ids` cannot be read as text. Names the file; a document that could not be read also carries a `reason`. |
| `file_too_large` | A file named in `file_ids` exceeds a size cap. Names the file, with `bound`, `limit`, and `tokens`. |
| `image_not_supported` | A file named in `file_ids` on this turn is an image, and the tier serving the turn does not read images. Names the image and the tier, and lists the tiers that do in `vision_tiers`. |
| `moderation_rejected` | The server-side input moderation gate refused the request's text. Carries `hazard_codes`. Not billed. |
| `pii_rejected` | A memory or rule contains an email address, phone number, or card number. Rephrase without it and retry. |
| `rules_too_large` | A ruleset exceeds a size cap. Names the cap. |
| `input_not_countable` | `POST /api/v1/tokens/count` refuses to count this text and returns no count. |
| `invalid_version` | The `Cruxy-Version` header is not a `YYYY-MM-DD` date. Answered on every route, before authentication. |
| `unknown_version` | `Cruxy-Version` is a well-formed date this server does not support. The message lists the supported dates. Answered on every route, before authentication. |
| `invalid_payload` | `POST /api/v1/feedback` only: the payload does not match the schema for its type. |
| `provider_not_supported` | Native sign-in named a provider that is not supported. |
| `redirect_uri_not_allowed` | Native sign-in named a redirect URI that is not allowlisted. |
| `scope_unsupported` | Retired: never sent. It stays in the vocabulary so the string is never reused. |
### 401 Unauthorized
| `code` | Meaning |
| --- | --- |
| `unauthorized` | The Bearer credential is missing, malformed, unknown, expired, or revoked, or its holder no longer belongs to the organization it names. Each case gets the same answer, with `WWW-Authenticate: Bearer`. It is the only `401` code; see [Get your API key](/docs/en/get-started/api-keys#expiry-and-revocation). |
### 403 Forbidden
The credential is genuine, but it cannot make this call. Waiting never widens a
credential, so retrying cannot succeed.
| `code` | Meaning |
| --- | --- |
| `account_suspended` | The key's owner's account is suspended. |
| `credential_quarantined` | The organization the key belongs to could not be established, so the key is refused rather than trusted. Signing in again does not clear it; contact support. |
| `forbidden_bucket` | The credential's billing kind cannot be billed on this route. No key issued today can reach it. |
| `key_scope_forbidden` | The key is not scoped for this endpoint family, or for the memory, rules, or files feature a request field asked for. Names `required_scope` in the second case. Re-scope or re-issue the key. |
| `model_not_allowed` | The key's model allowlist excludes the requested tier. |
| `key_read_only` | The key may not perform billable operations. |
| `org_forbidden` | The operation needs an organization owner or admin. |
| `workspace_forbidden` | Workspace management needs an organization owner or admin. |
| `logs_forbidden` | Request logs need an organization owner or admin. |
| `rules_forbidden` | You may not read or change this ruleset. |
| `vault_forbidden` | You take part in the vault but lack the permission the operation needs. |
| `conversation_forbidden` | The conversation is in a vault you may read but not write. |
### 404 Not Found
The target does not exist, or you cannot see it. The API gives the same answer
for an id that is missing and one you cannot see, so a `404` never confirms
that an id exists.
| `code` | What was not found |
| --- | --- |
| `user_not_found` | A user. |
| `file_not_found` | A file of yours under this key's organization, including one named in `file_ids`. |
| `request_not_found` | A `request_id` from a call this account made, on `POST /api/v1/feedback`. |
| `memory_not_found` | A memory. |
| `vault_not_found` | A vault you take part in. |
| `vault_member_not_found` | A vault member. |
| `org_member_not_found` | An organization member. |
| `invitation_not_found` | An invitation addressed to you that is still open. |
| `workspace_not_found` | A workspace in your organization. |
| `log_not_found` | A request log in your organization. |
| `conversation_not_found` | A conversation you can see. |
| `message_not_found` | A message in the conversation. |
| `prompt_not_found` | A prompt in your workspace. |
| `prompt_version_not_found` | That version of the prompt. |
| `eval_suite_not_found` | An eval suite in your workspace. |
| `eval_run_not_found` | An eval run in your workspace. |
| `eval_case_not_found` | A case in that eval suite. |
### 409 Conflict
The request conflicts with existing state. Only `idempotency_conflict` has a
retry path.
| `code` | Meaning |
| --- | --- |
| `idempotency_conflict` | The `Idempotency-Key` was reused with a different body. Resend the original body to replay the stored response, or use a new key. |
| `vault_name_exists` | A vault with this name already exists in your organization. |
| `vault_external_id_conflict` | The `external_id` names a vault somebody else owns. |
| `vault_member_exists` | The user is already a member of the vault. |
| `org_member_exists` | The user is already a member of the organization. |
| `org_member_active` | The member has not left the organization. |
| `workspace_slug_exists` | A workspace with this slug already exists. |
| `workspace_not_empty` | The workspace holds, or has held, a credential, so it cannot be deleted. |
| `workspace_is_default` | The default workspace cannot be deleted. |
| `prompt_slug_exists` | A prompt with this slug already exists. |
| `eval_suite_slug_exists` | An eval suite with this slug already exists. |
### 413 Payload Too Large
| `code` | Meaning |
| --- | --- |
| `payload_too_large` | The upload or message is over its size cap. |
| `storage_quota_exceeded` | The upload would take your files past your total storage allowance. |
### 422 Unprocessable Content
| `code` | Meaning |
| --- | --- |
| `structured_output_unsupported` | The tier that served the request does not support constrained decoding for `response_format`. |
| `structured_output_unsatisfied` | The model's non-streaming output did not satisfy the requested JSON schema. The request is still billed. |
### 429 Too Many Requests
Every `429` carries `Retry-After`, and nothing is billed.
| `code` | Meaning |
| --- | --- |
| `rate_limited` | Too many requests: `reason` is `rate` (the per-minute request window) or `concurrency` (too many of your requests in flight at once). Retry after `Retry-After`. |
| `budget_exhausted` | A signed-in plan's monthly or 12-hour burst pool is spent, or `mira`'s monthly request cap. `mira_available` says whether the same request on `mira` would be served now. Clears at `reset_at`. |
| `key_spend_cap_exceeded` | The key's monthly spend cap is spent. Clears at `reset_at`. |
| `workspace_spend_cap_exceeded` | The workspace's monthly spend cap is spent. Clears at `reset_at`. |
| `credits_exhausted` | A headless key's credit pool is drained. Clears at `reset_at`. |
| `credits_unavailable` | A headless key's credit balance could not be read. Retry after `Retry-After` (2 seconds). |
| `headless_concurrency_limit` | Chat only: a headless key already has its maximum requests in flight. Retry after `Retry-After` (1 second). |
### 5xx Server errors
On chat, every model failure adds `tier`, and none is billed.
| `code` | Status | Meaning |
| --- | --- | --- |
| `internal_error` | `500` | An unexpected server fault, on any route. Retry with backoff. |
| `upstream_error` | `502` | On a model call, the model provider rejected the request the gateway sent, and a retry fails the same way. On native sign-in, the identity provider could not be reached or answered unusably; retry. |
| `upstream_overloaded` | `503` | The model is overloaded, and the gateway's own retries ran out. Retry with backoff. |
| `model_unavailable` | `503` | The tier is temporarily unavailable: it kept failing and is cooling down, or it failed again on this request after the gateway's retries. Retry with backoff. |
| `all_models_unavailable` | `503` | Chat only: `model: "auto"` was sent and every tier is unavailable. Retry with backoff. |
| `server_shutting_down` | `503` | The gateway restarted, for a deploy, before it answered. Retry with backoff. |
| `limits_unavailable` | `503` | A rate-limit or budget check could not be read, so the request was refused rather than served unmetered. Adds `dimension`. Retry after `Retry-After` (5 seconds). |
| `count_unavailable` | `503` | `POST /api/v1/tokens/count` cannot count for this tier, because the count could disagree with billing. It does not clear on a retry. |
| `upstream_timeout` | `504` | The model did not answer in time. With `stream: false` the gateway made one attempt and does not retry it; see [Timeouts](#timeouts). |
### Stream errors
A stream that fails after it opens has already sent its `200`, so the code
arrives in the stream, not the status:
- `POST /api/v2/chat` ends with an `error` event whose `code` is one of
`upstream_error`, `upstream_overloaded`, `upstream_timeout`,
`model_unavailable`, `server_shutting_down`, or `internal_error`.
- `POST /v1/chat/completions` ends with a `data: {"error": {...}}` frame.
- `POST /api/v1/chat` sends no error frame: the stream ends without
`data: [DONE]`; see [Streaming](/docs/en/build/streaming#done-is-the-only-success-signal).
### Codes in request logs
`client_closed_request` (`499`) records that the caller disconnected before the
gateway had an answer. Nobody receives it; it appears only in request logs.
## Next steps
- [Reliability](/docs/en/build/reliability): deadlines, retries, idempotency, and platform status together.
- [Request IDs and metadata](/docs/en/build/request-ids): find the `request_id` and quote it when you report a problem.
- [Rate limits and budgets](/docs/en/build/rate-limits): read your headroom before you hit a `429`.
- [Streaming](/docs/en/build/streaming): detect a stream that failed partway.
- [Get your API key](/docs/en/get-started/api-keys): fix `401` and `403` at the key.
================================================================================
Page: Rate limits and budgets
URL: https://learn.getcruxy.com/docs/en/build/rate-limits
Markdown: https://learn.getcruxy.com/docs/en/build/rate-limits.md
================================================================================
# Rate limits and budgets
How the gateway meters you: rate_limited versus budget_exhausted, the rate-limit headers, spend caps, and reading your live headroom.
Two separate controls can refuse a request with `429`: a **rate limit** on how
fast you send requests, and a **budget** on how much you spend. The body's
`code` tells you which one you hit, and they need different handling. Your
account's limits are shown in your Console, and `GET /api/v1/limits` returns
them, with live headroom, to your code.
## rate_limited or budget_exhausted
| | rate_limited | budget_exhausted |
| --- | --- | --- |
| What ran out | Request rate: too many requests in the current window. | Allowance: the plan's monthly cap, its 12-hour burst pool, or the monthly `mira` request cap is spent. |
| What to do | Back off and retry after `Retry-After`. | Stop retrying. No immediate retry succeeds. |
| Clears | When the rate window resets. | At `reset_at`, when the spent window resets. |
| Headroom | `rate_limits.chat` on `GET /api/v1/limits` | `token_pool` on `GET /api/v1/limits` |
Spend caps refuse with `429` too, under their own codes: a monthly cap on a
key (`key_spend_cap_exceeded`), a monthly cap on a workspace
(`workspace_spend_cap_exceeded`), and the credit pool (`credits_exhausted`).
Like `budget_exhausted`, they mean an allowance is spent, and they clear only
at `reset_at` or when the cap changes. Every `429` carries `Retry-After`; on
these codes and on `budget_exhausted` it counts down to `reset_at`, and
retrying sooner fails the same way.
`POST /api/v1/moderations` never returns `budget_exhausted`: moderation is
exempt from the subscription pool, so a spent allowance cannot switch it off.
It can still return `rate_limited` or a spend-cap code.
## Rate-limit headers
Metered responses carry the state of the bucket the route is metered on:
| Header | Meaning |
| --- | --- |
| `X-RateLimit-Limit` | Requests allowed per window. |
| `X-RateLimit-Remaining` | Requests left in the current window, never below `0`. |
| `X-RateLimit-Reset` | When the current window ends, as an absolute Unix time in seconds, not a duration. |
| `Retry-After` | On every `429`, the seconds to wait. On a budget or spend-cap refusal, the seconds until `reset_at`. |
The `X-RateLimit-*` headers always describe the per-minute request window, so
a budget refusal can still show requests remaining.
The `X-RateLimit-*` headers are absent on a `401`, because authentication runs
before the limiter, and on a `403` from the key check (`key_scope_forbidden`,
`key_read_only`). They are also absent when the limiter cannot reach its own
storage; that request is served unmetered rather than refused.
```bash title="curl"
# -D - prints the response headers; look for X-RateLimit-* and, on a 429, Retry-After.
curl -s -D - -o /dev/null https://api.cruxy.ai/api/v1/limits \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
// These clear only at reset_at; every other 429 clears after Retry-After.
const ALLOWANCE_SPENT = new Set([
'budget_exhausted',
'key_spend_cap_exceeded',
'workspace_spend_cap_exceeded',
'credits_exhausted',
])
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'mira',
stream: false,
messages: [{ role: 'user', content: 'Hello' }],
}),
})
const remaining = res.headers.get('X-RateLimit-Remaining')
const resetAt = res.headers.get('X-RateLimit-Reset') // Unix seconds
if (res.status === 429) {
const { code, reset_at, request_id } = await res.json()
const waitSeconds = Number(res.headers.get('Retry-After'))
if (ALLOWANCE_SPENT.has(code)) {
// No retry succeeds before reset_at.
console.log(`allowance spent: ${code}, resets at ${reset_at} (request_id ${request_id})`)
} else {
// rate_limited, or a transient headless check: wait Retry-After, then retry.
console.log(`${code}; retry in ${waitSeconds}s (request_id ${request_id})`)
}
} else if (remaining !== null) {
console.log(`${remaining} requests left until ${new Date(Number(resetAt) * 1000).toISOString()}`)
}
```
Pace yourself before you reach `0`: when `X-RateLimit-Remaining` runs low,
slow down until `X-RateLimit-Reset` rather than waiting for a `429`.
## Read your limits
`GET /api/v1/limits` returns the limits that apply to your key, with live
headroom. Every value comes from the same limiter and budget state that chat
enforces, so `remaining` is exactly what your next request is checked against.
The call costs nothing, writes nothing, and is rate-limited like any other
route.
```bash title="curl"
curl https://api.cruxy.ai/api/v1/limits \
-H "Authorization: Bearer $CRUXY_API_KEY"
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/limits', {
headers: { Authorization: `Bearer ${process.env.CRUXY_API_KEY}` },
})
if (!res.ok) throw new Error(`limits request failed: ${res.status}`)
const limits = await res.json()
const perKey = limits.rate_limits.chat.per_key
console.log(`chat: ${perKey.remaining} of ${perKey.limit} left this window`)
if (limits.token_pool) {
console.log(`token pool: ${limits.token_pool.state}`) // healthy, low, or blocked
}
```
Everything comes from the key you send; nothing is read from the request.
Which blocks appear depends on your billing `bucket`:
| Field | Meaning |
| --- | --- |
| `object` | `limits`. |
| `bucket` | How this key is billed, and so which kind of key it is: `subscription` (a signed-in credential, with `token_pool`), `apikey` (a developer API key, billed per token), or `headless` (billed from prepaid `credits`). Memory and standing rules are on by default only for `subscription`. |
| `metered` | `true` for the `apikey` bucket only, and omitted otherwise: no token pool applies, and every request is billed per token. A key or workspace spend cap still binds it. |
| `tier` | Your account's subscription plan, which sizes `token_pool`. This is not a model tier. |
| `rate_limits.chat.per_key` | The chat request-rate window for this key: `limit`, `remaining`, `resets_at`, and `window_seconds`. `resets_at` is `null` until a request opens the window. |
| `rate_limits.chat.per_org` | The same window, for your whole organization. |
| `rate_limits.chat.max_concurrent_requests` | How many chat requests your account may have in flight at once, across all three chat endpoints. One more is refused at once, not queued: `429` `rate_limited` with `reason: concurrency` and `Retry-After: 1`. |
| `token_pool` | Subscription bucket only. The weighted-token budget behind `budget_exhausted`. |
| `credits` | Headless bucket only. The dollar-credit pool: `granted`, `adjustment`, `used`, `remaining`, and `resets_at`. |
| `key_spend_cap` | Present only when the key has a monthly spend cap: `cap`, `spent`, `remaining`, and `resets_at`, in USD. |
| `workspace_spend_cap` | Present only when the key's workspace has a monthly spend cap, in the same shape. |
`token_pool` has two windows, each with `cap`, `used`, `remaining`,
`resets_at`, and a `state` of `healthy`, `low`, or `blocked`:
- `monthly`: the calendar-month cap.
- `burst`: the trailing 12-hour sliding cap.
It also carries an overall `state`, `blocked_models`, the `unit`
(`weighted_tokens`), and `mira`, a monthly request cap on the `mira` tier with
its own `cap`, `used`, `remaining`, `resets_at`, and `state`.
There is no tokens-per-minute limit and no per-model daily cap. The gateway
enforces neither, so neither appears here.
## Budgets and auto
A request with `model: "auto"` may be budget-downgraded to `mira`. Read
`cruxy.routed_tier` on the response to see the tier that served it, rather than
assuming the router's first choice; see
[Auto routing](/docs/en/build/auto-routing).
## Next steps
- [Errors and retries](/docs/en/build/errors-and-retries): every `429` code, and a retry loop that honours `Retry-After`.
- [Features overview](/docs/en/get-started/features): the tiers and `auto`.
- [Get your API key](/docs/en/get-started/api-keys): key restrictions, including spend caps.
================================================================================
Page: Usage and cost
URL: https://learn.getcruxy.com/docs/en/build/usage-and-cost
Markdown: https://learn.getcruxy.com/docs/en/build/usage-and-cost.md
================================================================================
# Usage and cost
Read the usage object on each response: token counts, billable and cached tokens, the shape of the cost object, and how a stream reports usage.
Every chat response reports what the call consumed in a `usage` object: its
token counts and a `cost` object. This page documents the shape of both. Cost
amounts and rates are specific to your account and shown in your Cruxy
Console, so the examples here use placeholders where those numbers go. Read
the real values from your own responses.
```bash title="curl"
# usage, with its cost object, is near the end of the response body.
curl https://api.cruxy.ai/api/v1/chat \
-H "Authorization: Bearer $CRUXY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vaani",
"messages": [
{ "role": "user", "content": "Name three monsoon fruits." }
]
}'
```
```ts title="TypeScript"
const res = await fetch('https://api.cruxy.ai/api/v1/chat', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'vaani',
messages: [{ role: 'user', content: 'Name three monsoon fruits.' }],
}),
})
if (!res.ok) throw new Error(`chat failed: ${res.status} ${await res.text()}`)
const { usage, cruxy } = await res.json()
console.log(`request ${cruxy.request_id} on ${usage.cost.tier}`)
console.log(`input ${usage.input_tokens} (billable ${usage.billable_input_tokens})`)
console.log(`output ${usage.output_tokens}`)
// cost is an object, not a number: amount is the one figure, in cost.currency.
console.log(`cost ${usage.cost.amount} ${usage.cost.currency}`)
```
## The usage object
A non-streaming `POST /api/v1/chat` response carries `usage` next to `message`.
`input_tokens`, `output_tokens`, `billable_input_tokens`, and `cost` are always
present.
| Field | Meaning |
| --- | --- |
| `input_tokens` | The call's input tokens. |
| `output_tokens` | The tokens the model generated. |
| `billable_input_tokens` | The input tokens the call is billed for. |
| `cache_read_tokens` | Input tokens read from the prompt cache. |
| `cache_write_tokens` | Input tokens written to the prompt cache. |
| `prompt_tokens_details` | The same cache split in OpenAI's shape, `cached_tokens` and `cache_write_tokens`. Present only when the prompt cache was used. |
| `cost` | What the call cost, as an object; see [The cost object](#the-cost-object). |
The token fields have different names on different endpoints. Each set belongs
to a wire contract that has already shipped, so read the one for the endpoint
and mode you called:
| Endpoint and mode | Token fields | Cost |
| --- | --- | --- |
| `POST /api/v1/chat`, `stream: false` | `input_tokens`, `output_tokens`, `billable_input_tokens`, `cache_read_tokens`, `cache_write_tokens` | `usage.cost` |
| `POST /api/v1/chat`, `stream: true` | `prompt_tokens`, `completion_tokens`, `total_tokens`, plus the three Cruxy counts when they are not zero | `usage.cost`, on the usage frame |
| `POST /api/v2/chat`, either mode | `input_tokens`, `output_tokens`, `billable_input_tokens`, `cache_read_tokens`, `cache_write_tokens` | `usage.cost` |
| `POST /api/v1/embeddings` and `POST /api/v1/rerank` | `prompt_tokens`, `total_tokens` | `cruxy.cost` |
| `POST /v1/chat/completions` | `prompt_tokens`, `completion_tokens`, `total_tokens`, and `prompt_tokens_details.cached_tokens` when the cache was used | Not in the response |
| `POST /api/v1/moderations` | Not in the response | Not in the response |
## The cost object
`cost` is an object, not a number. Read `cost.amount` when you want one figure.
```json
{
"tier": "vaani",
"amount": "",
"currency": "",
"input_unit": "",
"output_unit": "