LoRAX Files

🎉 Enhancements

• Add prefix caching by @tgaddair in https://github.com/predibase/lorax/pull/581
• Add Llava Next (VLM) by @tgaddair in https://github.com/predibase/lorax/pull/586
• Embedder Service v0 with FlashBert by @magdyksaleh in https://github.com/predibase/lorax/pull/385
• Added eager prefill option by @tgaddair in https://github.com/predibase/lorax/pull/524
• BERT NER support by @magdyksaleh in https://github.com/predibase/lorax/pull/531
• Preload adapters during init by @tgaddair in https://github.com/predibase/lorax/pull/543
• Add support for batching to embedder models by @tgaddair in https://github.com/predibase/lorax/pull/503
• Bert to gpu by @magdyksaleh in https://github.com/predibase/lorax/pull/507
• Add distilbert by @magdyksaleh in https://github.com/predibase/lorax/pull/508
• feat: return usage in ChatCompletionStreamResponse by @GirinMan in https://github.com/predibase/lorax/pull/506
• Added Gemma2 by @tgaddair in https://github.com/predibase/lorax/pull/530
• Move kv cache allocation to router to ensure correct block allocation by @tgaddair in https://github.com/predibase/lorax/pull/545
• Tokenize inputs in router by @tgaddair in https://github.com/predibase/lorax/pull/548
• Add support for Llama 3 rotary embeddings by @tgaddair in https://github.com/predibase/lorax/pull/551
• Apply chat template in router to properly validate input length by @tgaddair in https://github.com/predibase/lorax/pull/538
• Allow eager_prefill to be set in Helm chart by @bdalal in https://github.com/predibase/lorax/pull/557
• Support FP8 for Mistral by @ajtejankar in https://github.com/predibase/lorax/pull/559
• Support FP8 for LLaMa by @ajtejankar in https://github.com/predibase/lorax/pull/562
• Support classify batch by @magdyksaleh in https://github.com/predibase/lorax/pull/577
• Adding longrope for serve Phi-3 by @huytuong010101 in https://github.com/predibase/lorax/pull/576
• Add new agnostic health endpoint by @magdyksaleh in https://github.com/predibase/lorax/pull/588
• Support FlashInfer for BERT by @tgaddair in https://github.com/predibase/lorax/pull/597
• Speed up NER inference by @magdyksaleh in https://github.com/predibase/lorax/pull/598
• Disable healthcheck tracing and add metrics to classify + classify_batch endpoints by @magdyksaleh in https://github.com/predibase/lorax/pull/603
• Added launcher args for preloaded_adapter_source and backend by @tgaddair in https://github.com/predibase/lorax/pull/604
• Parallelize tokenization for /classify_batch and remove block allocator for non-causal LMs by @tgaddair in https://github.com/predibase/lorax/pull/609
• support bge-base-en-v1.5 by @magdyksaleh in https://github.com/predibase/lorax/pull/593

🐛 Bugfixes

• Fix for the LM_HEAD issue by @ajtejankar in https://github.com/predibase/lorax/pull/475
• fix: load tokenizer/config with trust_remote_code by @thincal in https://github.com/predibase/lorax/pull/476
• Fix issue with Medusa batch load signature by @tgaddair in https://github.com/predibase/lorax/pull/492
• add missed dtypes for 8bit kv cache by @flozi00 in https://github.com/predibase/lorax/pull/490
• Fix quant cache OOM by @flozi00 in https://github.com/predibase/lorax/pull/494
• Add retries on common session errors for the client by @gyanesh-mishra in https://github.com/predibase/lorax/pull/495
• Revert AWQ to stable commit by @tgaddair in https://github.com/predibase/lorax/pull/498
• Fixed phi-3 with Su Rotary Embedding by @tgaddair in https://github.com/predibase/lorax/pull/499
• Fixed case where loaded lora adapter has no segments by @tgaddair in https://github.com/predibase/lorax/pull/510
• fix batching bug by @magdyksaleh in https://github.com/predibase/lorax/pull/513
• Fix issue with GQA initialization for Qwen2 by @arnavgarg1 in https://github.com/predibase/lorax/pull/514
• Disable fp8 kv cache for lovelace by @tgaddair in https://github.com/predibase/lorax/pull/520
• Bug fix for illegal memory access error caused when running medusa lora and plain loras in parallel. by @ajtejankar in https://github.com/predibase/lorax/pull/525
• bug : fix the type checking errors thrown by new ruff version by @ajtejankar in https://github.com/predibase/lorax/pull/533
• bug : fix Qwen-2 sliding_window config bug by @ajtejankar in https://github.com/predibase/lorax/pull/532
• Infer dtype from model config when not explicitly specified by @arnavgarg1 in https://github.com/predibase/lorax/pull/534
• Fix gemma2 by @Infernaught in https://github.com/predibase/lorax/pull/539
• Fix : compile bug causing models to error with 'lora' key not found by @ajtejankar in https://github.com/predibase/lorax/pull/547
• Fix: short circuit download, load, offload for preloaded adapters by @tgaddair in https://github.com/predibase/lorax/pull/552
• Fix the attention bug caused by upgrading vLLM by @ajtejankar in https://github.com/predibase/lorax/pull/555
• Fix LM head interaction with Medusa by @tgaddair in https://github.com/predibase/lorax/pull/567
• Fix adapter mask when using speculative decoding + LM head LoRA by @tgaddair in https://github.com/predibase/lorax/pull/570
• Fix outlines compatibility with speculative decoding by @tgaddair in https://github.com/predibase/lorax/pull/578
• Fix qwen lora by @magdyksaleh in https://github.com/predibase/lorax/pull/585
• Fix classify and classify_batch for Python client by @tgaddair in https://github.com/predibase/lorax/pull/608
• Fix ner entity merging by @magdyksaleh in https://github.com/predibase/lorax/pull/596
• Fix class ner by @magdyksaleh in https://github.com/predibase/lorax/pull/602
• Fix dependencies to address high urgency dependabot alerts by @magdyksaleh in https://github.com/predibase/lorax/pull/612

📝 Docs

• docs: update development_env.md by @eltociear in https://github.com/predibase/lorax/pull/515
• Doc updates for Medusa training by @arnavgarg1 in https://github.com/predibase/lorax/pull/544
• Add "pbase" to adapter_source docstrings by @alexsherstinsky in https://github.com/predibase/lorax/pull/583
• Add prerequisites to readme by @csabakecskemeti in https://github.com/predibase/lorax/pull/584

🔧 Maintenance

• chore: update infer.rs by @eltociear in https://github.com/predibase/lorax/pull/487
• start porting latest tgi by @flozi00 in https://github.com/predibase/lorax/pull/480
• Bump client to v0.6.1 by @tgaddair in https://github.com/predibase/lorax/pull/496
• Update Makefile-awq by @flozi00 in https://github.com/predibase/lorax/pull/493
• hqq upgrades by @flozi00 in https://github.com/predibase/lorax/pull/491
• try out an integration test workflow by @noyoshi in https://github.com/predibase/lorax/pull/516
• no warm up by @magdyksaleh in https://github.com/predibase/lorax/pull/540
• Update PyTorch, CUDA, vLLM, and Bitsandbytes by @ajtejankar in https://github.com/predibase/lorax/pull/553
• Added missing nvidia-ml-py package by @tgaddair in https://github.com/predibase/lorax/pull/558
• parse headers for errored requests by @noyoshi in https://github.com/predibase/lorax/pull/564
• handle folders for predibase by @noyoshi in https://github.com/predibase/lorax/pull/565
• enable mistral nemo by @magdyksaleh in https://github.com/predibase/lorax/pull/568
• bump version by @noyoshi in https://github.com/predibase/lorax/pull/569
• Install flashinfer in Docker by @tgaddair in https://github.com/predibase/lorax/pull/582
• feat : use --no-cache-dir flag to pip in dockerfiles to save space by @Rajpratik71 in https://github.com/predibase/lorax/pull/587
• Add missing configs by @magdyksaleh in https://github.com/predibase/lorax/pull/590
• Address rust compiler warnings by @magdyksaleh in https://github.com/predibase/lorax/pull/589

New Contributors

• @eltociear made their first contribution in https://github.com/predibase/lorax/pull/487
• @ajtejankar made their first contribution in https://github.com/predibase/lorax/pull/475
• @bdalal made their first contribution in https://github.com/predibase/lorax/pull/557
• @Rajpratik71 made their first contribution in https://github.com/predibase/lorax/pull/587
• @csabakecskemeti made their first contribution in https://github.com/predibase/lorax/pull/584

Full Changelog: https://github.com/predibase/lorax/compare/v0.10.0...v0.11.0