# LiteLLM (/docs/coreai/components/litellm)



Component Category [#component-category]

LLM proxy and gateway

Component Description [#component-description]

LiteLLM provides a unified gateway for accessing language models and model providers through a consistent interface.

Why It Is Used [#why-it-is-used]

It simplifies how CoreAI services call models, apply shared access patterns, and manage model routing and credentials across different providers and endpoints.

Learn More [#learn-more]

* [LiteLLM Documentation](https://docs.litellm.ai/docs/)
* [LiteLLM on GitHub](https://github.com/BerriAI/litellm)

Position In CoreAI [#position-in-coreai]

LiteLLM is an important internal CoreAI dependency, but it should not usually be treated as the primary long-term integration surface for product developers. The preferred stable product-facing contract is the `CoreAI API`.

Deployment notes [#deployment-notes]

* Requests through the gateway route and the proxy share a 600-second budget, so long generations and streaming responses are not cut off by shorter infrastructure timeouts.
* A dedicated Redis StatefulSet (`litellm-redis`) backs a response cache and keeps virtual-key authentication state consistent across workers and replicas.
* Worker count is pinned per pod, replicas spread across nodes, and failed upstream calls retry automatically.
* The image is mirrored in the private registry (version 1.89.4 as of platform 1.2.1).

Interacts With [#interacts-with]

* `CoreAI Web Portal` for user-facing model access
* `CoreAI API` for application-level LLM calls
* `Model Installer` for shared model access patterns
* `Redis` (`litellm-redis`) for response caching and shared authentication state
* `OpenTelemetry` and `MLflow` for tracing and GenAI observability
* Runtime inference services such as `vLLM` and `KubeAI`
* External model providers where applicable