LoRAX Files

🎉 Enhancements

• Prompt prefix caching for multi-LoRA by @tgaddair in https://github.com/predibase/lorax/pull/655
• Convert to Triton Punica kernels by @tgaddair in https://github.com/predibase/lorax/pull/658
• Support FP8 KV Cache by @ajtejankar in https://github.com/predibase/lorax/pull/652
• Added Mllama by @tgaddair in https://github.com/predibase/lorax/pull/619
• Flash mllama by @tgaddair in https://github.com/predibase/lorax/pull/622
• support MRL embeddings for qwen2 by @magdyksaleh in https://github.com/predibase/lorax/pull/621
• Support for Embeddings with XLM-RoBERTa and Adapters by @jfhetzer in https://github.com/predibase/lorax/pull/656
• Merge weights by @magdyksaleh in https://github.com/predibase/lorax/pull/600
• feat: Function calling with output schema enforcement by @jeffreyftang in https://github.com/predibase/lorax/pull/536
• Chunked prefill by @tgaddair in https://github.com/predibase/lorax/pull/653
• add num inputs to metrics by @magdyksaleh in https://github.com/predibase/lorax/pull/615
• Add --predibase-api-token CLI arg by @joseph-predibase in https://github.com/predibase/lorax/pull/617
• Add --disable-sgmv flag by @joseph-predibase in https://github.com/predibase/lorax/pull/639
• Enhance Structured Output Interface by @GirinMan in https://github.com/predibase/lorax/pull/644

🐛 Bugfixes

• Add done message to openai endpoints by @magdyksaleh in https://github.com/predibase/lorax/pull/618
• Fix CUDA graph compilation by @tgaddair in https://github.com/predibase/lorax/pull/627
• Fix CUDA graphs for Medusa by @tgaddair in https://github.com/predibase/lorax/pull/628
• Fix retrace message by @tgaddair in https://github.com/predibase/lorax/pull/629
• Fix prefix plumbing and BGMV compiler dimensions by @tgaddair in https://github.com/predibase/lorax/pull/631
• Fix punica kernel compilation by @tgaddair in https://github.com/predibase/lorax/pull/632
• Fix FlashInfer when not using prefix caching by @tgaddair in https://github.com/predibase/lorax/pull/633
• Fix cuda graph tracing without lora ranks by @tgaddair in https://github.com/predibase/lorax/pull/634
• Added ranks 96 and 128 to BGMV kernel by @tgaddair in https://github.com/predibase/lorax/pull/630
• Look for language model lm head by @Infernaught in https://github.com/predibase/lorax/pull/640
• Return n choices for chat completions API by @tgaddair in https://github.com/predibase/lorax/pull/638
• Fix llava_next for llama 3.2 vision cross attention states by @tgaddair in https://github.com/predibase/lorax/pull/641
• Fix compile for qwen-2.5-32b by @tgaddair in https://github.com/predibase/lorax/pull/645
• Added backwards compatible field to OpenAI json_object API by @tgaddair in https://github.com/predibase/lorax/pull/648
• Fix PREDIBASE_API_TOKEN env var being thrown away by @joseph-predibase in https://github.com/predibase/lorax/pull/654
• Fix absent fp8_kv property on llama and qwen models by @ajtejankar in https://github.com/predibase/lorax/pull/662
• Fix seqlen bug for sliding window models like Mistral v0.1 by @ajtejankar in https://github.com/predibase/lorax/pull/660
• Fix sliding window + compile bug by @ajtejankar in https://github.com/predibase/lorax/pull/666

📝 Docs

• added metrics docs, updated links in main docs by @noyoshi in https://github.com/predibase/lorax/pull/663

🔧 Maintenance

• upgrade poetry by @magdyksaleh in https://github.com/predibase/lorax/pull/613
• Fix deps4 by @magdyksaleh in https://github.com/predibase/lorax/pull/614
• Remove LD_PRELOAD from Docker and improve error message by @tgaddair in https://github.com/predibase/lorax/pull/623
• add label to id this as a lorax image by @noyoshi in https://github.com/predibase/lorax/pull/626
• pass correct stuff to predibase-reporter by @magdyksaleh in https://github.com/predibase/lorax/pull/635
• try using arc runner for build by @noyoshi in https://github.com/predibase/lorax/pull/646
• change runner 2 by @magdyksaleh in https://github.com/predibase/lorax/pull/650

New Contributors

• @joseph-predibase made their first contribution in https://github.com/predibase/lorax/pull/617
• @jfhetzer made their first contribution in https://github.com/predibase/lorax/pull/656

Full Changelog: https://github.com/predibase/lorax/compare/v0.11.0...v0.12.0