Route smarter,Pay less.
Orange reads every request, works out how much reasoning it actually needs, and sends it to the cheapest model that can handle it. Local classification, ~21ms, no network hop. Change your base URL and nothing else.
Three gates before a model is called
Orange sits between your application and your model providers. Each gate can make the call cheaper, or skip it entirely.
Exact-match cache
Hashes the whole conversation plus the parameters that change what a valid answer is. A hit returns with no classification and no model call. Zero false positives by construction — a cached answer is only ever served to a byte-identical request.
Complexity classifier
Embeds the request locally and scores it against labelled examples to pick a tier. No network hop, no second container. Only genuinely borderline requests escalate to a cheap judge model — currently 9.3% of traffic.
Failover & governance
A model that times out, rate-limits or 5xxs falls down the cost ladder — never up. Streaming fails over before the first byte reaches your client. Per-request and per-key spend ceilings, and a circuit breaker per provider.
Stop this madness 🤦♂️, route intelligently
It’s a whole genre at this point. What are we doing...
Got the monthly bill. We spent four figures sending 'extract the date from this string' to a frontier model. Forty thousand times. I need to lie down.
Why does classifying a boolean take four seconds? Because we send everything to the biggest model we have, and nobody wants to be the person who changed it.
Wrote our own 'AI router'. It is a 400-line if/else on prompt length. It misroutes every Tuesday and nobody can explain why.
One line of code. Zero refactoring.
Same SDK, same request shape, same response shape, same streaming, same typed errors. Anything requiring a refactor doesn't get adopted.
import OpenAI from "openai";
const openai = new OpenAI({
// Just change the base URL to point to your LLMRouter instance
baseURL: "https://router.yourdomain.com/v1",
apiKey: process.env.OPENAI_API_KEY,
});
const response = await openai.chat.completions.create({
model: "auto", // The router decides the actual model
messages: [{ role: "user", content: "Extract date from: May 4th 2024" }]
});Frequently Answered Questions
No. Orange speaks the OpenAI chat-completions API exactly — request shape, response shape, streaming chunks, usage blocks and typed errors. Here is the only change you need:
baseURL: "http://localhost:8000/v1"