Download
Punica is a system designed to efficiently serve multiple LoRA-fine-tuned large language models within a shared GPU environment. LoRA is a parameter-efficient fine-tuning method that allows developers to adapt large pretrained models to specific tasks by adding lightweight adapter layers rather than retraining the entire model. Punica introduces a serving architecture that allows multiple LoRA adapters to share the same base model during inference, significantly reducing memory consumption and computational overhead. The system includes specialized CUDA kernels that enable batched GPU operations across different LoRA models simultaneously. This design allows a single GPU cluster to host many task-specific models while maintaining high throughput and minimal latency. The architecture also includes scheduling mechanisms that coordinate requests from multiple tenants and distribute workloads efficiently across available resources.
Features
- Multi-tenant serving system for LoRA-adapted language models
- Shared base model architecture reducing memory usage
- Custom CUDA kernels enabling efficient batched LoRA inference
- GPU scheduling system for high throughput request handling
- Support for hosting multiple task-specific adapters simultaneously
- Optimized architecture for scalable LLM inference services
Project Samples
