KubeAI

Get inferencing running on Kubernetes: LLMs, Embeddings, Speech-to-Text. KubeAI serves an OpenAI compatible HTTP API. Admins can configure ML models by using the Model Kubernetes Custom Resources. KubeAI can be thought of as a Model Operator (See Operator Pattern) that manages vLLM and Ollama servers.

Features

Drop-in replacement for OpenAI with API compatibility
Serve top OSS models (LLMs, Whisper, etc.)
Multi-platform: CPU-only, GPU, coming soon: TPU
Scale from zero, autoscale based on load
Zero dependencies (does not depend on Istio, Knative, etc.)
Chat UI included (OpenWebUI)
Operates OSS model servers (vLLM, Ollama, FasterWhisper, Infinity)
Stream/batch inference via messaging integrations (Kafka, PubSub, etc.)

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow KubeAI

KubeAI Web Site

Other Useful Business Software

Veeam Data Platform v13.1 - Get Your Free Trial

Secure by design, portable by default. Recover clean, fast, anywhere. Start a free trial.

Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.

Try it Free

Rate This Project

User Reviews

Be the first to post a review of KubeAI!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Related Categories

Go Large Language Models (LLM), Go LLM Inference Tool

Registered

2024-09-25

Similar Business Software

LM-Kit.NET

LM-Kit.NET is a complete local AI runtime for .NET that lets engineering teams ship AI-powered features without cloud dependencies, per-token costs, or data leaving the network. Most .NET AI integrations stop at inference. LM-Kit.NET covers the full range of capabilities production...

See Software
Gemini Enterprise Agent Platform

Gemini Enterprise Agent Platform is a comprehensive solution from Google Cloud designed to help organizations build, scale, govern, and optimize AI agents. It represents the evolution of Vertex AI, combining advanced model development with new capabilities for agent orchestration and...

See Software
Google AI Studio

Google AI Studio is a unified development platform that helps teams explore, build, and deploy applications using Google’s most advanced AI models. It brings text, image, audio, and video models together in one interactive playground. With vibe coding, developers can use natural language to...

See Software
Compute with Hivenet

Hivenet Compute gives you on-demand RTX 5090 and RTX Pro 6000 GPUs, plus vCPU instances, billed per second while they run. Hivenet owns and operates the infrastructure, with data centres in France, the UAE and the US, so EU teams can keep workloads and data in Europe. Use it for training and...

See Software
Runpod

Runpod offers a cloud-based platform designed for running AI workloads, focusing on providing scalable, on-demand GPU resources to accelerate machine learning (ML) model training and inference. With its diverse selection of powerful GPUs like the NVIDIA A100, RTX 3090, and H100, Runpod supports...

See Software
Anaconda

Anaconda is an AI-native development platform that helps teams move from experimentation to production with trusted open-source packages, governed environments, and production-grade orchestration. The platform provides a secure foundation for Python, data science, machine learning, and AI...

See Software