Local Gateway with Ollama

Project Files

name: lite-llm-ollama

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: lite-llm-ollama-proxy
    env_file:
      - .env
    command: ["--config", "/app/config.yaml", "--host", "0.0.0.0", "--port", "4000"]
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER:-litellm}:${POSTGRES_PASSWORD:-litellm_password}@postgres:5432/${POSTGRES_DB:-litellm}
      OLLAMA_API_BASE: http://ollama:11434
    ports:
      - "${LITELLM_PORT:-4001}:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml:ro
    depends_on:
      ollama-pull-primary:
        condition: service_completed_successfully
      ollama-pull-secondary:
        condition: service_completed_successfully
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    container_name: lite-llm-ollama-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-litellm}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-litellm_password}
      POSTGRES_DB: ${POSTGRES_DB:-litellm}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-litellm} -d ${POSTGRES_DB:-litellm}"]
      interval: 5s
      timeout: 5s
      retries: 10
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    container_name: lite-llm-ollama
    ports:
      - "${OLLAMA_PORT:-11435}:11434"
    volumes:
      - ollama_data:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      timeout: 5s
      retries: 12
    restart: unless-stopped

  ollama-pull-primary:
    image: ollama/ollama:latest
    container_name: lite-llm-ollama-pull-primary
    environment:
      OLLAMA_HOST: http://ollama:11434
    command: ["pull", "${OLLAMA_MODEL_PRIMARY:-qwen2.5:0.5b}"]
    depends_on:
      ollama:
        condition: service_healthy
    restart: "no"

  ollama-pull-secondary:
    image: ollama/ollama:latest
    container_name: lite-llm-ollama-pull-secondary
    environment:
      OLLAMA_HOST: http://ollama:11434
    command: ["pull", "${OLLAMA_MODEL_SECONDARY:-smollm2:135m}"]
    depends_on:
      ollama:
        condition: service_healthy
    restart: "no"

volumes:
  ollama_data:
  postgres_data:

---

What you're running

Four kinds of containers, no cloud credentials:

• LiteLLM proxy — OpenAI-compatible API and admin UI on port 4001
• Ollama — runs local open-source models
• Postgres — required for the LiteLLM admin UI
• Model pull jobs — download tiny Ollama models, then exit

Model aliases the gateway exposes: The dummy model exists to fail. That is the point — it lets you prove fallback works without waiting for a real outage.

---

What this is NOT

• Not a production deployment — no TLS, no secret manager, no rate limiting, no autoscaling
• Not a benchmark — models are intentionally tiny so the demo starts fast on a laptop
• Not a replacement for a managed provider — it's the learning shape that makes the managed provider easier to debug later

If you want the production shape, that's Phase 6 — Bedrock Prod Profile.

---

Deploy

1. 1 Use Download All in the Project Files section above to save all three files into a new folder, e.g. lite-llm-ollama/
2. 2 Copy env.example to .env and adjust the master key if you like
3. 3 Open a terminal in that folder

cp env.example .env

docker compose --env-file .env -f docker-compose.yml down --remove-orphans
docker compose --env-file .env -f docker-compose.yml up -d

Watch the containers come up:

docker ps -a

A healthy run should look roughly like this:

lite-llm-ollama-proxy             Up (running)
lite-llm-ollama                   Up (healthy)
lite-llm-ollama-postgres          Up (healthy)
lite-llm-ollama-pull-primary      Exited (0)
lite-llm-ollama-pull-secondary    Exited (0)

---

Verify the model aliases

set -a
source .env
set +a

curl "http://localhost:${LITELLM_PORT:-4001}/v1/models" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Expected — four aliases loaded from the config:

local-free-chat
local-tiny-chat
local-dummy-chat
gpt-4o-mini

---

Lab 1: Real model works

curl "http://localhost:${LITELLM_PORT:-4001}/v1/chat/completions" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-free-chat",
    "messages": [
      {"role": "user", "content": "Reply with one short sentence: LiteLLM local works."}
    ]
  }'

The end-to-end path proven by this call:

curl ──► LiteLLM ──► Ollama ──► qwen2.5:0.5b

---

Lab 2: Fallback on purpose

Now call the broken alias:

curl "http://localhost:${LITELLM_PORT:-4001}/v1/chat/completions" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-dummy-chat",
    "messages": [
      {"role": "user", "content": "Say fallback worked in one short sentence."}
    ]
  }'

local-dummy-chat points at ollama_chat/not-a-real-model, so the first hop always fails. LiteLLM then walks the configured fallback list:

router_settings:
  num_retries: 0
  fallbacks:
    - local-dummy-chat: [local-free-chat, local-tiny-chat]

A successful response shows the actual fallback model in the model field:

{
  "model": "ollama_chat/qwen2.5:0.5b",
  "choices": [ { "message": { "content": "Fallback worked." } } ]
}

num_retries: 0 is set deliberately for the learning path so failures move to fallback immediately. In production you might choose differently.

---

Lab 3: Open the LiteLLM admin UI

The UI is at:

http://localhost:4001/ui/

Use the LITELLM_MASTER_KEY value from .env as the password.

In the UI, the model health table may initially show:

Health Status: none
Last Check:    None
Last Success:  None

That doesn't mean the models are healthy or unhealthy — it means health checks haven't run yet. Trigger one from the UI. The real aliases should succeed. The dummy alias may fail when checked directly, while the API fallback test still recovers. Both behaviors are correct.

---

Common Issues

Pull containers stuck in Created

The Ollama daemon takes a moment to become healthy. The pull jobs wait on condition: service_healthy. Give it 10–30 seconds and re-run docker ps -a.

connection refused from LiteLLM to Ollama

docker compose logs ollama

If Ollama crashed during model download (out-of-memory on a small laptop), drop one of the models:

# In .env — remove the secondary model
OLLAMA_MODEL_SECONDARY=

Then restart.

Health checks all fail in the UI

The health check calls the underlying provider. For local-dummy-chat that returns an error because the model genuinely doesn't exist — that is expected. The API fallback path still works.

---

Next Steps

• Provider Smoke Tests — prove free-tier provider keys directly, before wiring the gateway