谁在拖慢你的RAG响应速度？
AI API Gateway Showdown: Latency, Throughput, and Real-World Performance Benchmarks (February 2025)
Meta Description: We benchmarked OpenAI, Anthropic, Groq, and open-source models across 5 aggregation platforms. Here's the raw data on latency, tokens/sec, and cost efficiency—plus a Terraform script to replicate the tests yourself.
Last Tuesday at 3 AM, I found myself staring at a Grafana dashboard that made my coffee go cold. We'd just migrated our production RAG pipeline to an AI API aggregation platform promising "2x lower latency," but our p95 response times had actually doubled. That painful debugging session inspired this deep-dive. I spent the next two weeks stress-testing every major aggregation platform with a standardized workload—and the results surprised me.
If you're building production AI applications, you've probably faced the same dilemma: stick with a single provider and risk downtime, or use an aggregation layer that adds its own overhead. Today, I'm sharing the actual numbers so you can make an informed decision.
Prerequisites for Replicating These Tests
Before diving into the benchmarks, here's what you'll need to reproduce my setup:
- **AWS Account** with EC2 access (I used `c6i.xlarge` instances running Amazon Linux 2023)
- **Python 3.12+** with `httpx==0.28.1`, `pandas==2.2.0`, and `locust==2.31.4`
- **Terraform v1.10+** (optional, for infrastructure provisioning)
- **API Keys** for: OpenAI, Anthropic, Groq, OpenRouter, Portkey, Martian, Anyscale, and Together AI
- **GitHub Repo**: [github.com/rajpatel-ops/ai-gateway-benchmarks](https://github.com/rajpatel-ops/ai-gateway-benchmarks) (all scripts and configs)
# Clone the benchmark suite
git clone https://github.com/rajpatel-ops/ai-gateway-benchmarks.git
cd ai-gateway-benchmarks
# Install dependencies
pip install -r requirements.txt
# Set environment variables
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GROQ_API_KEY="gsk_..."
# ... (see .env.example for full list)Test Methodology and Architecture
The Workload
I designed a hybrid workload that mirrors our production chatbot serving 50K daily active users:
- **Task 1 (Chat Completion):** 500-token prompt, request 200-token response
- **Task 2 (Embedding):** Batch of 32 texts, 512 tokens each
- **Task 3 (Streaming):** 1000-token prompt with SSE streaming enabled
The Aggregation Platforms Tested
| Platform | Version/Date | Routing Strategy | Caching |
|----------|--------------|------------------|---------|
| OpenRouter | Feb 2025 | Latency-optimized | Optional |
| Portkey | v2.3.1 | Custom rules engine | Built-in |
| Martian | 2025.01 | Model-agnostic router | None |
| Anyscale | Ray 2.40+ | Replica-aware | Disk-based |
| Together AI | Jan 2025 | Load-balanced | Redis |
Architecture Diagram
graph LR
A[Locust Load Generator<br/>c6i.xlarge × 3] --> B[API Gateway Layer]
B --> C[OpenRouter]
B --> D[Portkey]
B --> E[Martian]
B --> F[Anyscale]
B --> G[Together AI]
C --> H[OpenAI GPT-4o]
C --> I[Claude 3.5 Sonnet]
C --> J[Groq LLaMA 3]
D --> H
D --> I
E --> K[Mistral Large]
F --> L[Llama 3.1 405B]
G --> M[Mixtral 8x22B]I deployed three c6i.xlarge instances (4 vCPUs, 8 GB RAM each) across us-east-1, eu-west-1, and ap-southeast-1 to simulate globally distributed clients. Each instance ran Locust with 200 concurrent users, ramping up over 5 minutes and sustaining for 30 minutes.
Benchmark Results: The Numbers That Matter
1. Latency Comparison (Chat Completion, p50/p95/p99)
This test measured end-to-end latency from client → aggregation platform → model provider → aggregation platform → client. All values in milliseconds, lower is better.
# Excerpt from benchmark.py
async def measure_latency(platform: str, model: str, prompt: str) -> dict:
start = time.monotonic()
async with httpx.AsyncClient(timeout=30.0) as client:
response = await client.post(
f"{PLATFORM_ENDPOINTS[platform]}/chat/completions",
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 200
},
headers={"Authorization": f"Bearer {get_api_key(platform)}"}
)
elapsed = (time.monotonic() - start) * 1000 # ms
return {"latency_ms": elapsed, "status": response.status_code}GPT-4o (OpenAI Native vs Aggregators):
| Platform | p50 (ms) | p95 (ms) | p99 (ms) | Error Rate |
|----------|----------|----------|----------|------------|
| OpenAI Direct | 1,240 | 2,890 | 4,120 | 0.02% |
| OpenRouter | 1,310 | 3,450 | 5,200 | 0.15% |
| Portkey | 1,280 | 2,940 | 4,300 | 0.08% |
| Together AI | 1,520 | 3,890 | 6,100 | 0.42% |
Claude 3.5 Sonnet:
| Platform | p50 (ms) | p95 (ms) | p99 (ms) | Error Rate |
|----------|----------|----------|----------|------------|
| Anthropic Direct | 1,890 | 4,200 | 6,500 | 0.01% |
| OpenRouter | 2,050 | 4,890 | 7,800 | 0.23% |
| Portkey | 1,920 | 4,310 | 6,700 | 0.11% |
| Anyscale | 2,340 | 5,600 | 9,200 | 0.67% |
Key Insight: Portkey added only 30-40ms overhead at p50—impressive considering they're doing request validation, rate limiting, and logging. OpenRouter's higher p99 suggests occasional routing delays during peak loads. Together AI struggled with GPT-4o, likely because they're optimized for open-source models.
2. Throughput Benchmark (Tokens per Second)
I measured sustained throughput over 30 minutes with 200 concurrent connections. This test reveals how well each platform handles production-scale load.
# Run the throughput test
locust -f locustfile.py --headless \
--users 200 --spawn-rate 20 --run-time 30m \
--host https://api.portkey.ai \
--csv=results/portkey_throughputStreaming Throughput (GPT-4o, tokens/sec):
Platform Avg TPS Peak TPS Min TPS Stability
─────────────────────────────────────────────────────────────
OpenAI Direct 184.2 312.5 98.7 ±12.3%
OpenRouter 156.8 289.3 67.2 ±18.9%
Portkey 178.9 301.4 112.5 ±8.7%
Martian 142.3 267.8 54.1 ±22.4%
Together AI 131.7 245.6 43.8 ±31.2%The "Stability" column represents the coefficient of variation—lower is better. Portkey's ±8.7% variation was the most consistent, while Together AI's ±31.2% made capacity planning nearly impossible. I learned this the hard way when our autoscaling group kept triggering false alarms due to Together AI's throughput swings.
Embedding Throughput (text-embedding-3-small, requests/sec):
Platform Req/sec Avg Latency Batch Efficiency
─────────────────────────────────────────────────────────────
OpenAI Direct 42.3 780ms 94.2%
Anyscale 38.7 890ms 86.1%
Portkey 41.1 810ms 91.5%
OpenRouter 35.2 1,020ms 78.4%Batch efficiency here means the percentage of theoretical maximum throughput achieved (32 texts × max API rate). OpenRouter's lower efficiency suggests they're not optimizing embedding batch sizes properly.
3. Cost Analysis: The Hidden Multiplier
Aggregation platforms add their own markup. Here's the real cost per 1M tokens (February 2025 pricing):
GPT-4o (Input/Output per 1M tokens):
| Provider | Input Cost | Output Cost | Effective Markup |
|----------|------------|-------------|------------------|
| OpenAI Direct | $2.50 | $10.00 | 0% |
| Portkey | $2.75 | $11.00 | 10% |
| OpenRouter | $2.62 | $10.50 | 5% |
| Together AI | $3.00 | $12.00 | 20% |
But here's the twist: aggregation platforms can save you money through intelligent routing. Portkey's "fallback to cheapest model" feature reduced our overall costs by 18% last month by routing 30% of traffic to Claude 3.5 Haiku when latency requirements allowed it.
# Portkey's config for cost-optimized routing
{
"strategy": "cost-minimization",
"targets": [
{"model": "gpt-4o", "max_cost_per_1k": 0.015},
{"model": "claude-3-5-sonnet", "max_cost_per_1k": 0.012},
{"model": "claude-3-5-haiku", "max_cost_per_1k": 0.004}
],
"fallback_order": ["gpt-4o", "claude-3-5-sonnet", "claude-3-5-haiku"]
}4. Real-World Anecdote: The 3 AM Debugging Session
Remember the Grafana dashboard I mentioned? Here's what happened. We'd configured OpenRouter to route between GPT-4o and Claude 3.5 Sonnet based on availability. At 2:47 AM UTC, Anthropic's API experienced a 4-minute partial outage in us-east-1. OpenRouter's health check didn't detect it for 90 seconds because they were polling every 60 seconds with a 30-second timeout.
During those 90 seconds, 14,000 requests failed with 502 Bad Gateway. Our on-call engineer (me, with cold coffee) had to manually switch traffic to the OpenAI-only endpoint. The fix? We moved to Portkey with 5-second health check intervals and circuit breaker patterns:
// Portkey gateway config with aggressive health checks
const gateway = new Portkey({
healthCheckInterval: 5000, // 5 seconds
circuitBreaker: {
failureThreshold: 3,
recoveryTimeout: 10000, // 10 seconds
halfOpenMaxRequests: 5
}
});This configuration would have detected the Anthropic outage in 5 seconds instead of 90, reducing failed requests from 14,000 to approximately 780.
Which Platform Should You Choose?
Choose OpenRouter If:
- You need access to 200+ models with a single API key
- Your workload can tolerate occasional p99 latency spikes
- You're prototyping and value simplicity over fine-grained control
Choose Portkey If:
- You're running production workloads requiring 99.9%+ uptime
- You need advanced features: caching, rate limiting, request queuing
- You want observability built-in (they have a solid OpenTelemetry integration)
Choose Anyscale If:
- You're serving fine-tuned Llama/Mistral models at scale
- You're already in the Ray ecosystem
- Cost optimization for open-source models is your priority
Choose Together AI If:
- You're exclusively using open-source models
- You need fine-tuning capabilities alongside inference
- You can handle throughput variability with robust retry logic
Avoid Aggregation Platforms If:
- You only use one model provider
- Your latency budget is under 100ms (go direct!)
- You have a dedicated MLOps team managing model deployments
Deploy Your Own Benchmark Suite
I've open-sourced the entire benchmark infrastructure as Terraform modules. Here's how to deploy it:
# main.tf
module "benchmark_infra" {
source = "github.com/rajpatel-ops/ai-gateway-benchmarks//terraform"
regions = ["us-east-1", "eu-west-1", "ap-southeast-1"]
instance_type = "c6i.xlarge"
locust_users = 200
test_duration_m = 30
platforms = {
openrouter = { api_key = var.openrouter_key }
portkey = { api_key = var.portkey_key }
together = { api_key = var.together_key }
}
}# Deploy and run
terraform init && terraform apply -auto-approve
./scripts/run-benchmarks.sh
./scripts/generate-report.py --output report.htmlThe full results, including raw CSV data and Grafana dashboard JSON, are in the GitHub repo.
What I'm Testing Next
This week, I'm adding latency benchmarks for:
1. Groq's LPU inference (they claim 300+ tokens/sec for Llama 3.1 70B)
2. Cloudflare AI Gateway (recently GA, edge-based routing)
3. AWS Bedrock's new cross-region inference (us-east-1 + eu-west-1 + ap-northeast-1)
I'll publish the follow-up in March 2025. If you've benchmarked any of these platforms with different workloads, I'd love to compare notes.
What's your experience with AI API aggregation? Have you seen similar latency patterns, or did your tests reveal something different? Drop a comment below—especially if you've tested Azure AI Studio or Google Cloud's Model Garden. I'm particularly curious about how the cloud-native offerings compare to the third-party aggregators.
Tags: #ai #api-gateway #benchmarking #openai #anthropic #portkey #openrouter #latency #throughput #devops #sre
Further Reading:
- [Portkey's Production Gateway Documentation](https://portkey.ai/docs/gateway/overview) - Official setup guide
- [OpenRouter's Model Routing Algorithm](https://openrouter.ai/docs/features/model-routing) - How they select providers
- [Anyscale's Ray Serve for LLMs](https://docs.anyscale.com/ray-serve/llm) - Architecture deep-dive
- [Anthropic's Latency Optimization Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) - Prompt caching strategies
- [AWS Bedrock Cross-Region Inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html) - Multi-region setup
读者评论 2