一个API标准意外治好了我们的长尾模型焦虑

Experience: How we built an LLM gateway to aggregate long-tail model traffic by riding on OpenAI’s API standard

TIL that 70% of our GPU costs were going to models nobody had heard of, and our infra team was about to revolt. Fair warning: this is a war story about how we accidentally solved the long-tail model problem by just copying what OpenAI did with their API spec.

So here’s the situation. Our company runs an internal ML platform for various teams—marketing wants LLaMA for copy generation, legal needs Claude for contract review, and some PhD in R&D keeps requesting obscure models from HuggingFace that have like 47 downloads total. For a while we just spun up dedicated endpoints for each one. That worked until it didn’t.

Picture this: you’re on pager duty at 2 AM because a niche embedding model from a Chinese university is OOM-ing on a single A100 that’s otherwise sitting at 3% utilization. I think the model was called something like bge-large-zh-v1.5—actually wait, I should clarify that it was specifically the v1.3 release from December 2023 that had the memory leak. v1.5 was fine. But nobody had bothered to update the deployment config because the original requester had left the company six months ago.

Meanwhile, our GPT-4 proxy is getting absolutely hammered and we can’t route around it because every model has its own bespoke API format. I distinctly remember our lead infra engineer—this is Dave, who’s been doing SRE since before Kubernetes was a thing—saying “I don’t care if it’s GPT-5 from the future, if it doesn’t speak /v1/chat/completions it’s dead to me.”

That rant turned into our architecture north star.

The ugly reality of long-tail models

First, some numbers to ground this. We audited our model usage over 30 days (this was February 2024):

• Top 3 models (GPT-4, Claude-3, LLaMA-70B) accounted for 82% of requests
• The remaining 18% was spread across **37 different models**
• Some of those models got literally 12 requests per day. Twelve.
• We were maintaining 41 unique deployment configs, auth patterns, and retry logic per model

The long tail was killing us on operational overhead, not compute. Every weird model meant another Docker container, another set of environment variables, another thing that could break during a deploy. And don’t get me started on streaming—some models used SSE, some used WebSockets, one even returned newline-delimited JSON without the data: prefix. I wish I was making that up. The commit message when we discovered that was just “why.”

The OpenAI API as an unintentional standard

Here’s the thing I’ve noticed after lurking on r/MachineLearning for years: whether you love or hate OpenAI, their API format has become the de facto standard. It’s like REST for LLMs. Every major inference engine (vLLM, TGI, TensorRT-LLM) now supports it natively. Even Anthropic’s SDK can be configured to speak OpenAI-format with a translation layer.

So we thought: what if we just built a gateway that makes every model look like an OpenAI endpoint? Then anything downstream that expects POST /v1/chat/completions with a standard payload just works™.

I know this sounds obvious in retrospect. It really does. But the key insight was doing this at the aggregation layer rather than wrapping each model individually. Instead of 37 adapters, we built one translation engine that maps between the OpenAI spec and whatever weird format a long-tail model expects.

How we actually built it (the non-marketing version)

We used LiteLLM as the translation core. Yes, it’s open source, yes it has quirks—I think we hit a bug with their Gemini streaming in v1.28.3 that required a workaround—but it handles 100+ model providers and you can extend it without forking. The gateway sits behind a single URL: https://llm-gateway.internal/v1. Every team hits that endpoint with standard OpenAI payloads.

Under the hood:

1. Request lands → Gateway inspects the model field (e.g., "bge-large-zh")

2. Routing table lookup → Maps to the actual backend config (HuggingFace TGI, vLLM, etc.)

3. Payload translation → Converts OpenAI format to whatever the backend wants

4. Load balancing → Distributes across our GPU pool based on model affinity and current load

5. Response normalization → Takes the weird response and makes it look like OpenAI’s streaming or non-streaming format

The real magic is in the routing table. We built a simple config that lets anyone register a model:

models:
 - name: bge-large-zh
 backend: vllm
 endpoint: 10.12.44.7:8000
 max_batch_size: 32
 cost_center: legal-dept

Suddenly that obscure embedding model is just another model parameter. The gateway handles batching, retries, and auth. The legal team doesn’t know or care that it’s running on a dedicated node in our DC—they just see openai.Embedding.create(model="bge-large-zh", input=text).

Well... that’s the theory. In practice, we had to add a max_retries field per model after the incident I’m about to describe.

The numbers after 3 months

I’m not gonna pretend this was a flawless rollout. We had a spectacular outage on March 14th—I remember because it was Pi Day and I was supposed to leave early—when someone registered a model with a typo in the endpoint (10.12.44.7:800 instead of 8000) and the gateway’s retry logic went exponential. The error logs were just pages of ConnectionRefusedError growing at 2x per second. Pro tip: always set max_retries per model, not globally. We learned that one the hard way.

But once we stabilized:

• Infrastructure maintenance time dropped 60%
• GPU utilization went from 31% to 78%
• New model onboarding: from ~2 days to ~15 minutes
• We decommissioned 12 dedicated endpoints that were running at <5% utilization

The real win was organizational. Teams stopped asking “can you deploy model X for us?” and started asking “can I get an API key for the gateway?” It shifted us from being model janitors to platform builders. Dave actually smiled once. I think. It might have been gas.

The catch (there’s always a catch)

YMMV significantly. This approach works because OpenAI’s format is flexible enough to handle 90% of use cases. But if you need model-specific features—like Claude’s tool use before OpenAI supported it, or Gemini’s grounding—you’ll hit the limits of translation. We ended up with a hybrid approach: standard features go through the gateway, exotic stuff gets a dedicated endpoint. That’s like 5% of traffic now.

Also, cost attribution becomes critical. When everything looks like “gpt-4” to the client but it’s actually running on your own hardware, you need solid tracking. We built a per-token cost model into the gateway that tracks actual GPU time and maps it to department budgets. Without that, you’ll get a nasty surprise at the end of the quarter when finance asks why the “free” internal models cost $40K in electricity and hardware depreciation. Ask me how I know.

Why I’m posting this

I saw that thread last week about “are LLM gateways just API proxies” and the top comment was like > “it’s just nginx with extra steps lol.” That’s technically true in the same way that Kubernetes is just Docker with extra steps. The value isn’t in routing—it’s in making heterogeneous infrastructure look homogeneous to consumers.

If you’re dealing with more than 5 models, or if your team keeps asking for random HuggingFace checkpoints, do yourself a favor and standardize on the OpenAI API format at the edge. It’s not perfect, but it’s the least bad option we have right now. Probably.

TL;DR: Long-tail models create disproportionate operational overhead. We built a gateway that makes every model speak OpenAI’s /v1/chat/completions format, cutting maintenance by 60% and doubling GPU utilization. The trick is translating at the aggregation layer, not per-model. It’s not a silver bullet but it beats maintaining 37 bespoke deployments.

Anyone else doing something similar? Curious how you handle streaming edge cases—we still struggle with models that return tokens in weird chunk sizes. Had one model from a research lab in Singapore that would buffer exactly 7 tokens before flushing. Seven. Why seven? I still don’t know.

Edit: Thanks for the gold, kind stranger. To the commenter asking about LiteLLM vs building from scratch—we tried building our own translation layer first. Spent 3 weeks on it before realizing we were just badly reinventing LiteLLM. Don’t be like us. The moment I knew we’d messed up was when I found myself writing a regex to parse Anthropic’s streaming format at 11 PM on a Saturday. That’s a special kind of rock bottom.

Tags: #llm #mlops #gateway #openai-api #infrastructure #warstory #devops