optillm Files

What's Changed

Fix: runaway local generation for models whose ChatML end token differs from their tokenizer EOS (#317)

Running such a model through optillm's built-in local inference never stopped early and generated the full max_new_tokens default (4096) on every call that omitted max_tokens — e.g. dhara-250m's chat ends at <|im_end|> (49154) but its tokenizer eos_token is <|end_of_text|> (1), so it rambled ~800s/call.

• Honor generation_config.eos_token_id: generation now resolves EOS from the model's own generation config (merging the tokenizer EOS as a fallback) instead of hardcoding tokenizer.eos_token_id. Fixes any chat model with this mismatch; well-behaved models (e.g. Qwen2.5, whose tokenizer EOS already is <|im_end|>) are unchanged.
• OPTILLM_MAX_TOKENS: new env var to override the default max_new_tokens (default stays 4096), bounding any request that sends no max_tokens. An explicit request max_tokens still takes precedence. Useful for small models that don't reliably emit an EOS token.

Only the local inference engine (OPTILLM_API_KEY=optillm) is affected; proxied providers handle stopping themselves. Includes unit tests for both helpers.

Full Changelog: https://github.com/algorithmicsuperintelligence/optillm/compare/v0.3.19...v0.3.20