whichllm 0.6.8

vLLM and SGLang are in. A row of a repository's own Hugging Face weights, which whichllm ranks on Linux with an NVIDIA GPU and under WSL, now names the server that runs it and the command. That holds when the weights split over several GPUs too, and when an MoE pages its experts in from RAM.

vLLM and SGLang

AWQ, GPTQ, FP8, NVFP4, MXFP4, INT8 and BF16 repos had no engine and no run line before. Now such a row names vLLM with vllm serve <repo> --max-model-len auto, or SGLang with sglang serve <repo> when SGLang is the one running or the only one installed. The cards are written from vLLM 0.33.0 and SGLang 0.5.23. On NVIDIA both need Linux, compute capability 7.5 or newer and a driver for CUDA 13, and both run under WSL. (#310)

These rows used to take llama.cpp's fit and speed, with MoE experts or whole layers in RAM, and neither server runs them that way. Their RAM offload streams the weights on every token, 2.9 to 16.6 tok/s on a 16 GB card in SGLang's own measurements. So a row of Hugging Face weights now shows only where vLLM or SGLang hold it, and the model's GGUF covers the rest.

Several GPUs

Weights too big for one GPU now split evenly over 2, 4 or 8 GPUs with tensor parallelism, and the run line adds --tensor-parallel-size N for vLLM or --tp-size N for SGLang. Every card holds the same share, so the smallest card in the group bounds the split, and three cards use two. On four 4090s, 13 rows of the top 60 now name vLLM with TP=4, among them Qwen3.5-122B-A10B GPTQ-Int4 at 27.8 tok/s. Discussion [#110] asked for this. (#311)

The speed stays at a layer split's for now, which undersells it. A community test of vLLM with TP=2 on two 3090s ran Llama 3.1 70B AWQ at 21.4 tok/s, and whichllm gives Llama 3.3 70B AWQ 10.1 tok/s there. That is one measurement, so the gain stays out until there are more. If you run vLLM or SGLang over several GPUs, a tok/s figure in [#110] would help.

Paged Experts

An MoE past one GPU's VRAM in BF16, FP16, FP8, AWQ, GPTQ, NVFP4 or MXFP4 can now run with SGLang's Paged Experts. SGLang keeps K of each layer's experts on the GPU and pages the rest in from RAM, and the row shows as GPU+RAM. K is what SGLang's static budget, 0.88 of the GPU, leaves after the other weights, the KV cache and the CUDA context, as SGLang's docs work it out. The run line passes it with --paged-experts-num-resident K --mem-fraction-static 0.88. For Qwen3-30B-A3B GPTQ-Int4 on a 16 GB card it comes to 92, the value the docs use on that card.

Every expert stays pinned in RAM, which WSL caps at about half the RAM. The speed is llama.cpp's for the same split, and SGLang's own measurements match or beat it at one request. (#311)

whichllm doesn't check yet which architectures each server supports, and pipeline parallelism isn't modeled.

Windows on a WSL share

whichllm on Windows took itself for WSL when it ran from a folder on a WSL share, \\wsl.localhost\.... There Windows resolves /proc/version to the distro's own file, which says "microsoft". So whichllm engines said Abyss and Project Maya do not run under WSL, and the ranking left out their packs. Only Linux is checked for WSL now. (#307)

How it was checked

• I compared the top 60 with 0.6.7 for a simulated RTX 3060, 4060 Ti, 5080, 4090 and 5090 and for two 3090s, two 4090s and four 4090s. The top 15 is the same on all eight. Below that, rows of Hugging Face weights that fit only with part of them in RAM are gone, one or two on one GPU and four on two GPUs, all from 23rd down.
• New tests cover the engine choice, the run lines, the even split, K and its flags, the WSL pinned limit and the new card fields. All 2,005 tests pass.

Upgrade

:::bash
uv tool upgrade whichllm
pip install -U whichllm
brew upgrade whichllm

Everything else is in CHANGELOG.md (github.com).