Building a Free-Tier Multi-Provider LLM Gateway with LiteLLM: A Complete Setup Guide

Overview

This guide walks through setting up a self-hosted LiteLLM proxy that unifies multiple free-tier LLM providers (NVIDIA NIM and OpenRouter) behind a single OpenAI-compatible API endpoint. It also covers automated model-availability probing and integrating the resulting gateway with an AI coding CLI.

No API keys, account names, personal identifiers, or real secrets are included anywhere in this guide. All example values are placeholders — replace them with your own credentials.

Why build this

• Free-tier LLM providers often enforce low per-minute rate limits (e.g., 20-40 requests/minute).
• Model catalogs change frequently — models get added, deprecated, or temporarily rate-limited.
• A single unified gateway lets any OpenAI-compatible client (coding CLIs, IDE extensions, custom scripts) fail over automatically across multiple providers/models without code changes.

Part 1: Probing Model Availability

Before wiring a provider into a gateway, verify which models are actually reachable with your account/key at that moment. A good probe script should:

• Dynamically fetch the live model catalog via the provider's /models endpoint rather than hardcoding model names (catalogs change).
• Send a minimal streaming request (e.g., "Reply with exactly: OK") and mark a model usable as soon as the first real content/reasoning delta arrives — no need to wait for a full completion.
• Respect the provider's documented rate limit with a safety margin (e.g., use 15 RPM if the documented limit is 20 RPM).
• On an HTTP 429 (rate limited), skip only that specific model and continue testing the rest, rather than aborting the whole run.
• Log every outcome — success, timeout, HTTP error body, network exception — to a structured file (CSV/JSONL) for later review.
• Exclude non-chat model types automatically: embeddings, rerankers, text-to-speech, content-safety/moderation classifiers, and image/video generators are not useful in a general chat/coding pool.
• Never print or log the API key. Load it from a local .env file (excluded from version control), and apply a redaction filter to all log output and error payloads in case the key ever appears inside an exception message.

Run the probe with a conservative rate and a small batch size first, then scale up once you confirm authentication and behavior are correct:

python probe_models.py --max-models 10 --rpm 8

The output should include a plain-text list of confirmed-usable model IDs, plus a CSV with per-model status, HTTP code, latency, and any error detail — this becomes your source of truth for the gateway configuration, rather than assuming every catalog entry is currently reachable.

Part 2: Building the LiteLLM Gateway

LiteLLM is an open-source proxy that exposes a single OpenAI-compatible API (/v1/chat/completions) and routes requests to dozens of underlying providers. Key building blocks:

docker-compose.yml (minimal, no database)

services:
  litellm:
    image: ghcr.io/berriai/litellm:<pinned-version>
    env_file:
      - ./litellm.env
    ports:
      - "127.0.0.1:4000:4000"
    environment:
      LOG_LEVEL: INFO
    volumes:
      - ./litellm_config.yml:/app/config.yaml:ro
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    restart: unless-stopped

Binding to 127.0.0.1 keeps the proxy accessible only from the local machine. Pin the image to a specific version tag rather than latest to avoid unexpected breaking changes between runs.

litellm.env (secrets file — never commit this)

PROVIDER_A_API_KEY=replace-with-your-real-key
PROVIDER_B_API_KEY=replace-with-your-real-key
LITELLM_MASTER_KEY=replace-with-a-long-random-secret-starting-with-sk-
UI_USERNAME=replace-with-a-username
UI_PASSWORD=replace-with-a-strong-password

Add this file to .gitignore immediately. The master key is what OpenAI-compatible clients send as their Bearer token — it must start with sk-.

litellm_config.yml (model routing)

model_list:
  - model_name: free-tier-pool
    litellm_params:
      model: openai/<provider-model-id-1>
      api_base: https://provider-endpoint.example/v1
      api_key: os.environ/PROVIDER_A_API_KEY
      order: 1
      max_tokens: 8192
      timeout: 120

  - model_name: free-tier-pool
    litellm_params:
      model: openai/<provider-model-id-2>
      api_base: https://provider-endpoint.example/v1
      api_key: os.environ/PROVIDER_A_API_KEY
      order: 2
      max_tokens: 8192
      timeout: 90

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 0
  allowed_fails: 1
  cooldown_time: 300

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Important routing notes:

• Multiple entries sharing the same model_name form a load-balanced failover group — clients requesting that name get whichever deployment the router selects, based on the order field and current health/cooldown state.
• Order deployments from fastest/most-capable-when-healthy to slowest/most-specialized. Put anything with high observed latency near the bottom.
• Set cooldown_time generously (e.g., 5 minutes) so a temporarily rate-limited deployment isn't retried immediately, wasting more of the shared quota.
• If you want a client to explicitly target one specific underlying model rather than "whatever the pool picks," give that deployment its own unique model_name in addition to (or instead of) the shared pool entry.

The api_key field must use the exact env-var reference syntax

A common failure mode: the master key reference (os.environ/LITELLM_MASTER_KEY) fails to resolve at config-parse time in some container/version combinations, producing authentication and "not connected to DB" errors even when the underlying key value is correct. If this happens, temporarily hardcode the literal secret value directly in the config file as a diagnostic step, confirm the proxy comes up healthy, then decide whether to keep the direct value (acceptable for a strictly local, loopback-only deployment) or continue debugging the environment-variable resolution path for a more portable setup.

Part 3: Common Pitfalls and Fixes

Symptom	Root cause	Fix
Read timeout on every non-streaming request	Client used a blocking (non-streaming) call against a provider/model with high time-to-first-token	Switch health checks to streaming mode; mark success on first delta, not full completion
"LLM Provider NOT provided" error	Missing or malformed adapter prefix in the model string	Use <adapter>/<exact-provider-model-id> consistently for every entry
HTTP 404 mentioning data-retention/guardrail violation	Account-level or workspace-level zero-data-retention policy excludes providers that don't support it	Review and adjust privacy/guardrail settings if you're comfortable with the trade-off, understanding it may route prompts to providers with different data policies
HTTP 402 insufficient credits on an "auto" router model	Automatic/task-aware routers can select paid underlying models even when marketed as free-to-call	Exclude auto-routing meta-models from strictly free-tier pools; use explicit free-only router variants instead
HTTP 429 with "worker limit reached" or "shared pool" messaging	Upstream provider's shared free capacity is temporarily saturated by other users	Not fixable client-side; add cooldown + failover so the gateway automatically tries the next deployment
HTTP 403 "only available via agentic harness"	Some free models are gated to recognized coding-agent integrations and reject raw API probes	Exclude these from generic health-check scripts; they may still work inside a supported agent/IDE integration
"Unknown command" from the proxy container	Overriding the container's entrypoint command incorrectly (e.g., wrapping it in a shell one-liner not supported by the image)	Use the image's documented CLI argument list format directly in the compose command field

Part 4: Connecting a Coding CLI

Most modern AI coding CLIs support custom OpenAI-compatible providers via a local config file, typically requiring three things:

1. A base URL pointing at your gateway (e.g., http://localhost:4000/v1).
2. An environment variable name the CLI reads the bearer token from — never paste the key directly into the CLI's config file.
3. A wire/API protocol setting — confirm whether your CLI expects the Chat Completions format or the newer Responses format, since gateways may only fully support one.

Test with a minimal one-off prompt before committing the configuration permanently, and only switch to your default model-name once you've confirmed the round trip works end-to-end.

Security Checklist

• Never commit .env/secrets files to version control — add them to .gitignore before your first commit.
• Bind the proxy to 127.0.0.1 unless you specifically need LAN/remote access, and if you do, put it behind authentication and TLS.
• Rotate any API key immediately if it is ever pasted into a chat log, screenshot, terminal recording, or shared document.
• Use a long, randomly generated master key — not a short or guessable string.
• If you enable a proxy's admin web UI, create a dedicated admin account rather than relying indefinitely on a shared environment-variable credential.

Summary

A local LLM gateway built on an open-source proxy lets you combine several free-tier providers into one resilient, OpenAI-compatible endpoint. The reliability of the whole system depends on: probing real-time model availability rather than trusting static documentation, ordering fallback deployments by observed latency and reliability, and handling provider-side rate limits gracefully with cooldowns instead of hard failures.