inference-gateway

README

Inference Gateway

The Inference Gateway is a proxy server designed to facilitate access to various language model APIs. It allows users to interact with different language models through a unified interface, simplifying the configuration and the process of sending requests and receiving responses from multiple LLMs, enabling an easy use of Mixture of Experts.

• Key Features
• Supported API's
• Configuration
• Examples
• License

Key Features

• 📜 Open Source: Available under the MIT License.
• 🚀 Unified API Access: Proxy requests to multiple language model APIs, including Groq, OpenAI, Ollama etc.
• ⚙️ Environment Configuration: Easily configure API keys and URLs through environment variables.
• 🐳 Docker Support: Use Docker and Docker Compose for easy setup and deployment.
• ☸️ Kubernetes Support: Ready for deployment in Kubernetes environments.
• 📊 OpenTelemetry Tracing: Enable tracing for the server to monitor and analyze performance.
• 🛡️ Production Ready: Built with production in mind, with configurable timeouts and TLS support.
• 🌿 Lightweight: Includes only essential libraries and runtime, resulting in smaller size binary of ~10.8MB.
• 📉 Minimal Resource Consumption: Designed to consume minimal resources and have a lower footprint.
• 📚 Documentation: Well documented with examples and guides.
• 🧪 Tested: Extensively tested with unit tests and integration tests.
• 🛠️ Maintained: Actively maintained and developed.
• 📈 Scalable: Easily scalable and can be used in a distributed environment - with HPA in Kubernetes.
• 🔒 Compliance and Data Privacy: This project does not collect data or analytics, ensuring compliance and data privacy.
• 🏠 Self-Hosted: Can be self-hosted for complete control over the deployment environment.

Supported API's

• OpenAI
• Ollama
• Groq Cloud
• Google
• Cloudflare

Configuration

The Inference Gateway can be configured using environment variables. The following environment variables are supported.

Examples

• Using Docker Compose
• Using Kubernetes
• Using standard REST endpoints

License

This project is licensed under the MIT License.