LiteLLM

Component Category

LLM proxy and gateway

Component Description

LiteLLM provides a unified gateway for accessing language models and model providers through a consistent interface.

Why It Is Used

It simplifies how CoreAI services call models, apply shared access patterns, and manage model routing and credentials across different providers and endpoints.

Learn More

• LiteLLM Documentation
• LiteLLM on GitHub

Position In CoreAI

LiteLLM is an important internal CoreAI dependency, but it should not usually be treated as the primary long-term integration surface for product developers. The preferred stable product-facing contract is the CoreAI API.

Deployment notes

• Requests through the gateway route and the proxy share a 600-second budget, so long generations and streaming responses are not cut off by shorter infrastructure timeouts.
• A dedicated Redis StatefulSet (litellm-redis) backs a response cache and keeps virtual-key authentication state consistent across workers and replicas.
• Worker count is pinned per pod, replicas spread across nodes, and failed upstream calls retry automatically.
• The image is mirrored in the private registry (version 1.89.4 as of platform 1.2.1).

Interacts With

• CoreAI Web Portal for user-facing model access
• CoreAI API for application-level LLM calls
• Model Installer for shared model access patterns
• Redis (litellm-redis) for response caching and shared authentication state
• OpenTelemetry and MLflow for tracing and GenAI observability
• Runtime inference services such as vLLM and KubeAI
• External model providers where applicable