Download Latest Version v2.2.0_ Falcon, macOS support, and more.zip (160.0 kB)
Email in envelope

Get an email when there's a new version of Petals

Home / v2.0.1
Name Modified Size InfoDownloads / Week
Parent folder
README.md 2023-07-23 2.8 kB
0
v2.0.1_ Inference of longer sequences, Python 3.11 support, bug fixes.tar.gz 2023-07-23 99.1 kB
0
v2.0.1_ Inference of longer sequences, Python 3.11 support, bug fixes.zip 2023-07-23 140.8 kB
0
Totals: 3 Items   242.8 kB 0

Highlights

🛣️ Inference of longer sequences. We extended the max sequence length to 8192 tokens for Llama 2 and added chunking to avoid server out-of-memory errors (happened when processing long prefixes). This became possible thanks to multi-query attention used in Llama 2, which uses 8x less GPU memory for attention caches. Now you can process longer sequences using a Petals client and have dialogues of up to 8192 tokens at https://chat.petals.dev

🐍 Python 3.11 support. Petals clients and servers now work on Python 3.11.

🐞 Bug fixes. We fixed the server's --token argument (used to provide your 🤗 Model Hub access token for loading Llama 2), possible deadlocks in the server, issues with fine-tuning speed (servers available via relays are deprioritized) and other minor load balancing issues.

🪟 Running server on Windows. We made a better guide for running a server in WSL (Windows Subsystem for Linux).

📦 Running server on Runpod. We added a guide for using a Petals template on Runpod.

What's Changed

Full Changelog: https://github.com/bigscience-workshop/petals/compare/v2.0.0.post1...v2.0.1

Source: README.md, updated 2023-07-23