Load Testing and Capacity Planning for the Gateway
The gateway is on the hot path — k6 / Locust against the proxy, profiling the Postgres and Redis ceilings, deciding where horizontal scaling starts to matter.
💡
Tip: Most teams skip load testing the gateway until the first incident. The honest version of this use case is the one that runs the test before that incident.
What this use case covers
- k6 and Locust scripts pointed directly at the LiteLLM proxy, with realistic prompt mixes.
- Profiling the Postgres ceiling — the admin DB is usually the first thing to wobble.
- Profiling the Redis ceiling — cache lookups under load are deceptively expensive.
- Deciding where horizontal scaling starts to matter: one big container, three medium ones, or a real autoscaling group.
- A short capacity-planning conversation template you can take to the platform leadership meeting.