Moderation gate

Screen user input with cruxy-guard before it reaches chat, and act on the flagged boolean, not the uncalibrated category scores.

A moderation gate runs Moderations in front of Chat: classify the input first, and only call the model if it comes back clean. This recipe wires the two together and shows the one decision the guard's response forces you to get right.

The gate

Classify the prompt with cruxy-guard. If the single result is flagged, refuse before spending a chat call; otherwise pass the prompt through to chat.

const BASE = 'https://api.cruxy.ai'
const auth = {
  Authorization: `Bearer ${process.env.CRUXY_API_KEY}`,
  'Content-Type': 'application/json',
}
 
async function screen(text: string): Promise<{ flagged: boolean; requestId: string }> {
  const res = await fetch(`${BASE}/api/v1/moderations`, {
    method: 'POST',
    headers: auth,
    body: JSON.stringify({ model: 'cruxy-guard', input: text }),
  })
  const body = await res.json()
  return {
    flagged: body.results[0].flagged,
    // Moderations carries no cruxy block; the request id is the top-level
    // `id`, prefixed "modr-". Strip the prefix to get the bare request_id.
    requestId: body.id.replace(/^modr-/, ''),
  }
}
 
async function guardedChat(prompt: string): Promise<string> {
  const gate = await screen(prompt)
  if (gate.flagged) {
    console.warn(`blocked input, request_id=${gate.requestId}`)
    return 'This request was blocked by the content filter.'
  }
 
  const res = await fetch(`${BASE}/api/v1/chat`, {
    method: 'POST',
    headers: auth,
    body: JSON.stringify({
      model: 'kavi',
      stream: false,
      messages: [{ role: 'user', content: prompt }],
    }),
  })
  const { message } = await res.json()
  return message.content
}

Branch on flagged, never on the scores

The guard returns category_scores alongside the categories booleans, and it is tempting to threshold them. Do not:

// WRONG: category_scores is not a calibrated probability.
if (result.category_scores.violence > 0.7) block()

The guard model emits labels, not likelihoods, so every value in category_scores is exactly 1.0 or 0.0, the same bit as the matching boolean in categories, written as a number. A threshold like 0.7 buys nothing: nothing ever lands between the two values, and reading a score as confidence reads a precision that is not there. Decide on flagged for an any-category gate, or on a specific categories[name] boolean when you only want to block one kind of content. The Moderations guide carries the same caveat for the endpoint on its own.

Screening a batch or the output

input also accepts an array, and results returns one entry per input in order; screen every message in a conversation with one call rather than one call each. The gate reads left to right, so you can run the same screen over the model's reply before you show it, and refuse on the way out as well as on the way in.

Next steps

• Moderations: the categories, the response, and the binary-scores caveat.
• Chat: the endpoint the gate protects.
• Errors: the 400, 401, and 429 bodies.