Foundations

Course Roadmap & Gateway Pattern

OpenAI-Compatible API as the Stable ContractSoon

Provider Economics in 2026Soon

LiteLLM vs Cloudflare, Kong, Portkey, Self-BuiltSoon

Course Setup, Repo Layout & Four-Curl MethodSoon

Production-Grade Is a Checklist, Not a BinarySoon

Local-First With Ollama

LiteLLM + Ollama with Fallback

The Four-Curl Debugging Method, AppliedSoon

Why Postgres Ends Up in the Local StackSoon

Multi-Provider Routing

Direct Provider Validation (Pre-Gateway)Five Providers, One Endpoint

Routing Strategies Deep DiveSoon

Declared Fallback Chains in YAMLSoon

Per-Request Fallback OverrideSoon

Streaming & the Mid-Stream Fallback LimitSoon

Provider-Specific GotchasSoon

Enterprise Cloud — Bedrock (Dev)

Why the Dev Profile Exists SeparatelySoon

Smallest Verifiable Bedrock Setup

Seven Bedrock Failure Modes by LikelihoodSoon

Bedrock Chat — Nova, Claude Haiku, Claude SonnetSoon

Bedrock Embeddings — Titan & CohereSoon

Bedrock Streaming & IAM PermissionSoon

Aliasing Bedrock Models to OpenAI DefaultsSoon

Other Enterprise Providers

Azure OpenAI — Deployment Names & VersionsSoon

Google Vertex AISoon

Anthropic Direct (Non-Bedrock)Soon

Cohere, Mistral & Second-Tier ProvidersSoon

The Provider Matrix — When Each One WinsSoon

Production-Shape Platform (Bedrock Prod)

Three-Service Topology — Gateway + Postgres + RedisSoon

Healthchecks & depends_on: service_healthySoon

STORE_MODEL_IN_DB & the Admin UISoon

Production-Shape Bedrock Stack

The Checklist Framing & Gap TableSoon

Laptop → Real Server Promotion ChecklistSoon

IAM-Role-Ready AuthSoon

Cost Governance

Three Kinds of Keys — Master, Operator, AppSoon

Per-Key Budgets, TPM/RPM & Model ScopingSoon

Team-Scoped Model Access & Least PrivilegeSoon

Spend Tracking & Reconciliation GapsSoon

Pre-Call Estimation vs Post-Call ReconciliationSoon

Showback & ChargebackSoon

The Finance Conversation in Engineer-SpeakSoon

Caching

Exact-Match Redis CacheSoon

Semantic Cache with Similarity ThresholdSoon

Cache Invalidation — TTL, Flush, Version-KeyedSoon

Measuring the Delta — p50/p95, Cost, Hit RateSoon

Honest Cache Limits per Workload ShapeSoon

Observability

What to Log at the Gateway LayerSoon

Shipping Logs to Loki / Datadog / CloudWatchSoon

Prometheus Exporter & Four Standard DashboardsSoon

Langfuse for Trace-Level VisibilitySoon

OpenTelemetry ExportSoon

Alerting on Cliffs That MatterSoon

The On-Call Runbook for an AI GatewaySoon

Guardrails, PII & Compliance

PII Detection & Redaction (Presidio)Soon

Prompt-Injection Detection InboundSoon

Content Filtering OutboundSoon

Per-Key Guardrail PoliciesSoon

SOC2 / HIPAA / GDPR PostureSoon

Audit Logging for Regulated WorkloadsSoon

Data Residency, Made ConcreteSoon

Authentication & Multi-Tenancy

Bearer Tokens vs JWT ValidationSoon

OIDC — Okta, Auth0, Cognito, Azure ADSoon

End-User Identity → Virtual Key MappingSoon

SaaS Multi-Tenancy Key PatternsSoon

BYOK — Bring-Your-Own-KeySoon

Per-Tenant Budgets, Model Access, AuditSoon

Advanced Routing

A/B Testing & Shadow TrafficSoon

Canary Deployments for Model SwapsSoon

Latency-Aware RoutingSoon

Cost-Aware RoutingSoon

Circuit Breakers for Provider OutagesSoon

Multi-Region / Multi-Account FailoverSoon

Blast-Radius Isolation Across AccountsSoon

Beyond Text

Embeddings + Vector DB (pgvector / Qdrant / Pinecone)Soon

Full RAG Loop Behind the GatewaySoon

Image Models — DALL-E, Stable DiffusionSoon

Audio — Whisper, ElevenLabs, PollySoon

Vision & Multimodal ParitySoon

Tool / Function-Calling Parity Across ProvidersSoon

Self-Hosted Models at Scale

vLLM as a Serving BackendSoon

Text Generation Inference (TGI)Soon

GPU Autoscaling (KEDA, Karpenter)Soon

Model Swap Without DowntimeSoon

Hybrid Cloud + On-Prem Behind One GatewaySoon

Emerging Standards

Model Context Protocol (MCP) HubSoon

Agent Frameworks Behind the GatewaySoon

Realtime / WebSocket APIsSoon

Streaming Function-Call Wire FormatsSoon

CI/CD & Config-as-Code

LiteLLM Config Schema Validation in CISoon

Dry-Run Testing for Model-List ChangesSoon

Staged Rollout — Dev → Staging → ProdSoon

GitOps with Flux / ArgoCDSoon

Disaster Recovery and BackupsSoon

Rolling Back a Bad Model Swap in < 5 MinSoon

Load Testing & Capacity

Why the Gateway Is on the Hot PathSoon

k6 & Locust Scripts Against the ProxySoon

Profiling the Postgres CeilingSoon

Profiling the Redis CeilingSoon

When Horizontal Scaling Starts to MatterSoon

Capacity-Planning ConversationSoon

Capstone — Production End-to-End

The Brief — B2B SaaS AI FeatureSoon

Full Stack — TLS + RDS + ElastiCache + ObsSoon

IAM Model & Audit TrailSoon

Deployment — ECS / EKS / Single EC2Soon

Load Test, Failover Drill, Rollback DrillSoon

90-Day Operational ReviewSoon

Submitting the Capstone ArtifactsSoon

Career & Communication

Positioning as an LLM Platform EngineerSoon

System Design Interviews for AI GatewaysSoon

Writing the Right Kind of Public ContentSoon

The About Blurb That Signals AuthoritySoon

Building a Public Portfolio RepoSoon

Use Cases — Publishing & Video Arc

MCP (Model Context Protocol) Routing Through the Gateway

Emerging standard in late 2025/early 2026 — the gateway as an MCP hub, fronting tools and context providers the same way it fronts models.

💡

Tip: MCP is the most interesting new protocol in the LLM space since the OpenAI Chat API. The gateway as MCP hub is the natural next layer.

What this use case covers

A short MCP primer: what the protocol is, what problem it solves, why it is showing up everywhere in 2026.
The gateway as an MCP hub: tools, resources, and context providers registered once, available to every client.
Per-key MCP policies: which tools and resources each virtual key can reach.
The honest gaps while the protocol stabilizes — what works today and what will change.

Why it matters

MCP turns "every agent reimplements every tool" into "every agent points at the same MCP hub." The gateway is the obvious place for that hub to live, because it already owns identity, budgets, and audit.

Coming soon

Implementation coming soon. This use case will ship as a runnable folder in infinite-containers/lite-llm, with a step-by-step lesson here and a companion post and video, once the protocol stabilizes further. Track the full publishing roadmap in USE-CASES.md.