The first version of any multi-model chatbot has the routing logic smeared
across the UI. A provider === "openai" branch here, a streaming special-case
there, a prompt template inlined into a click handler. It works — right up
until you add the third model.
Push the decision behind one seam
The move that paid off most was refusing to let the client know which model answered. It sends a query and a task hint; a single orchestration layer picks the provider, shapes the prompt, and normalises the stream back into one shape.
const route = pickModel(task); // "reasoning" → Gemini, "fast" → GPT‑mini
const stream = await route.invoke(prompt, { stream: true });
Everything provider-specific — auth, token limits, the quirks of each streaming format — lives on one side of that seam. The client just renders tokens as they arrive.
Good abstractions here aren't about hiding the models. They're about hiding the differences between them.
The parts worth standardising
| Concern | Why it belongs in the layer |
|---|---|
| Prompt shaping | Each model wants a slightly different frame |
| Streaming | Normalise chunks so the UI stays dumb |
| Fallbacks | One provider down shouldn't be an outage |
On streaming specifically
Keeping the response responsive matters more than which model you picked. If the first token lands fast and the stream never stalls, users forgive a lot. Buffer nothing you don't have to; flush as it comes.
The payoff isn't cleverness — it's that adding a fourth provider becomes a config change, not a refactor.