KubeAI

Get inferencing running on Kubernetes: LLMs, Embeddings, Speech-to-Text. KubeAI serves an OpenAI compatible HTTP API. Admins can configure ML models by using the Model Kubernetes Custom Resources. KubeAI can be thought of as a Model Operator (See Operator Pattern) that manages vLLM and Ollama servers.

Features

Drop-in replacement for OpenAI with API compatibility
Serve top OSS models (LLMs, Whisper, etc.)
Multi-platform: CPU-only, GPU, coming soon: TPU
Scale from zero, autoscale based on load
Zero dependencies (does not depend on Istio, Knative, etc.)
Chat UI included (OpenWebUI)
Operates OSS model servers (vLLM, Ollama, FasterWhisper, Infinity)
Stream/batch inference via messaging integrations (Kafka, PubSub, etc.)

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow KubeAI

KubeAI Web Site

Other Useful Business Software

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Rate This Project

User Reviews

Be the first to post a review of KubeAI!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Related Categories

Go Large Language Models (LLM), Go LLM Inference Tool

Registered

2024-09-25

Similar Business Software

LM-Kit.NET

LM-Kit.NET is a complete local AI runtime for .NET that lets engineering teams ship AI-powered features without cloud dependencies, per-token costs, or data leaving the network. Most .NET AI integrations stop at inference. LM-Kit.NET covers the full range of capabilities production...

See Software
Gemini Enterprise Agent Platform

Gemini Enterprise Agent Platform is a comprehensive solution from Google Cloud designed to help organizations build, scale, govern, and optimize AI agents. It represents the evolution of Vertex AI, combining advanced model development with new capabilities for agent orchestration and...

See Software
Google AI Studio

Google AI Studio is a unified development platform that helps teams explore, build, and deploy applications using Google’s most advanced AI models. It brings text, image, audio, and video models together in one interactive playground. With vibe coding, developers can use natural language to...

See Software
Runpod

Runpod offers a cloud-based platform designed for running AI workloads, focusing on providing scalable, on-demand GPU resources to accelerate machine learning (ML) model training and inference. With its diverse selection of powerful GPUs like the NVIDIA A100, RTX 3090, and H100, Runpod supports...

See Software
OpenRouter

OpenRouter is an AI model routing platform that gives developers access to hundreds of models through a single unified API. It connects users with models from providers such as OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, Qwen, xAI, and many others. The platform supports text, image,...

See Software
NVIDIA Triton Inference Server

NVIDIA Triton™ inference server delivers fast and scalable AI in production. Open-source inference serving software, Triton inference server streamlines AI inference by enabling teams deploy trained AI models from any framework (TensorFlow, NVIDIA TensorRT®, PyTorch, ONNX, XGBoost, Python,...

See Software