Oumi v0.8 Release Notes

This release lands the new oumi deploy CLI for shipping models to dedicated inference endpoints, an oumi-mcp server for MCP-capable assistants, batch-API parity across hosted providers, and a major dependency push to Transformers``v5 /TRL0.30+ /vLLM` 0.20+.

Highlights

oumi deploy — new CLI for dedicated inference endpoints

A first-class deployment CLI that uploads a model and stands up a dedicated endpoint on a managed inference provider. Fireworks.ai lands first, Parasail ships in this release, and the architecture (base_client.py, typed exception hierarchy) is built so additional providers can plug in.

What it does in one go: validates the model directory → uploads weights (full model or LoRA adapter) → creates a dedicated endpoint with the requested hardware → polls until live → optionally fires test prompts.

:::bash
# Single-command deploy from a YAML config
oumi deploy up --config configs/examples/deploy/fireworks_deploy.yaml

# Or assemble the deploy on the CLI
oumi deploy up \
  --model-path /path/to/my-finetuned-model/ \
  --provider fireworks \
  --hardware nvidia_h100_80gb \
  --gpu-count 2 \
  --min-replicas 1 \
  --max-replicas 4

# Lifecycle commands
oumi deploy upload     --model-path ... --provider fireworks --wait
oumi deploy create-endpoint --model-id ... --provider fireworks --wait
oumi deploy status     --endpoint-id ep-123 --provider fireworks --watch
oumi deploy list       --provider fireworks
oumi deploy list-models --provider fireworks
oumi deploy list-hardware --provider fireworks

Example fireworks_deploy.yaml:

:::yaml
model_source: /path/to/my-finetuned-model/
provider: fireworks
model_name: my-finetuned-model-v1
model_type: full          # or "adapter" for LoRA + base_model: ...
hardware:
  accelerator: nvidia_h100_80gb
  count: 2
autoscaling:
  min_replicas: 1
  max_replicas: 4
test_prompts:

  - "Hello, how are you?"

oumi-mcp — Model Context Protocol server

Oumi now ships an MCP server so MCP-capable assistants (Claude Desktop, Claude Code, Cursor, …) can browse the ~500 bundled YAML configs, launch and monitor training / eval / inference jobs locally or on cloud, and read built-in workflow guidance — all from chat.

:::bash
pip install "oumi[mcp]"
# Two equivalent entry points, both stdio:
oumi-mcp
python -m oumi.mcp

Wire it up in your client (Claude Code shown):

:::bash
claude mcp add oumi oumi-mcp

Or in claude_desktop_config.json / Cursor / ~/.claude.json:

:::json
{
  "mcpServers": {
    "oumi": { "command": "oumi-mcp" }
  }
}

Built-in prompts cover get-started, train, infer, eval, synth, analyze, post-training, cloud-launch, and an end-to-end mle_workflow. Full guide: docs/user_guides/mcp.md.

Batch API support across hosted inference providers

Hosted batch-inference parity. oumi infer and the engines now expose batch endpoints for Anthropic, Fireworks, and Together, plus job control (cancel, partial retry) and progress tracking.

:::python
from oumi.core.configs import InferenceConfig
from oumi.inference import AnthropicInferenceEngine

config = InferenceConfig.from_yaml("infer.yaml")
engine = AnthropicInferenceEngine(model_params=config.model)

# Submit, poll, fetch results — engine handles the batch lifecycle
results = engine.infer_batch(input=conversations, inference_config=config)

:::yaml
# Inside an inference YAML, opt in to the batch API
remote_params:
  use_batch_api: true
  batch_completion_window: "24h"

RPM / TPM rate limiting on RemoteInferenceEngine

Sliding-window rate limiting baked into every remote engine so you can pin API budgets without reaching for an external proxy. Tracks RPM, input TPM, and output TPM independently from each provider response. politeness_policy is now deprecated in favor of requests_per_minute.

:::yaml
# infer.yaml
model:
  model_name: claude-opus-4-7

engine: ANTHROPIC

remote_params:
  num_workers: 16
  requests_per_minute: 4000
  input_tokens_per_minute: 400_000
  output_tokens_per_minute: 80_000

Cerebras inference engine

New engine targeting Cerebras-hosted models, registered with the standard InferenceEngineType factory.

:::yaml
model:
  model_name: llama-3.3-70b
engine: CEREBRAS
remote_params:
  api_key_env_varname: CEREBRAS_API_KEY

:::python
from oumi.inference import CerebrasInferenceEngine
engine = CerebrasInferenceEngine(model_params=...)

Related work:

• Cerebras Inference Support (#2231)

Multi-turn conversation synthesis

oumi synth learned to chain conversation synthesizers into multi-turn dialogues — useful for distilling assistant/customer-support style training data.

:::bash
oumi synth -c oumi://configs/examples/synthesis/multiturn_conversation_synth.yaml

The bundled config generates 5 customer-support conversations with structured action blocks (CLARIFY, LOOKUP_ORDER, INITIATE_RETURN, …); use it as a starting point for your own scenarios. See docs/user_guides/synth.md for the synthesizer composition model.

Related work:

• Feature/multiturn synth (#2172)
• Add token usage accumulation to AttributeSynthesizer (#2201)
• Treat empty arrays in synthesis config as unset (#2246)

Judge framework upgrades

The Judge gained batch inference, token-usage accounting, and the ability to take pre-built Conversation objects (handy when you already constructed multi-turn context elsewhere).

:::python
from oumi.judge import judge_dataset

results = judge_dataset(
    judge_config="oumi://configs/judges/helpfulness.yaml",
    dataset=[
        {"question": "What is 2+2?", "answer": "4"},
        {"question": "How to cook?", "answer": "I don't know"},
    ],
    output_file="judgments.jsonl",
)
for r in results:
    print(r.field_values, r.field_scores)

Inference engine quality of life

Smaller but high-leverage improvements that show up everywhere remote inference is used:

• list_models() API on every engine — discover what a provider exposes from code (#2333).
• finish_reason is now surfaced on conversation outputs across engines (#2249).
• Quota errors are a typed, catchable error class instead of opaque HTTP failures (#2222).
• Transient HTTP 400s are retried with backoff (#2357).
• api_input is attached to APIStatusError for easier debugging (#2355).
• Empty Anthropic responses raise an explicit error (#2346); reasoning-model content: null is handled (#2354).
• Anthropic prompt caching enabled; cache token usage reported for Anthropic and Together (#2324, [#2245]).
• vLLM engine accepts arbitrary kwargs through vllm_config_overrides for late-breaking server flags (#2314).
• Step is now included in metrics-logger callback (#2244).
• Temperature constraints handled for all OpenAI reasoning models (#2212).

• Lambda inference engine deprecated (#2332).

  :::python engine.list_models(chat_only=True) # ['accounts/.../models/llama-v3p1-70b-instruct', ...]

  result = engine.infer(...) result[0].messages[-1].finish_reason # 'stop' | 'length' | 'tool_calls' | ...

Config + CLI error handling

Configuration errors used to surface as raw OmegaConf stack traces. They now flow through a typed exception hierarchy with explicit messages:

• OumiConfigParsingError covers all OmegaConf exceptions (#2323).
• Config-specific exception hierarchy + CLI error handling (#2319).

• DatasetParams.finalize_and_validate checks dataset_path exists (#2320).

  :::python from oumi.core.configs import TrainingConfig from oumi.exceptions import OumiConfigParsingError

  try: config = TrainingConfig.from_yaml("train.yaml") except OumiConfigParsingError as e: # Clean, user-facing error — not an OmegaConf traceback print(e)

oumi launcher cost & schedule fields

The launcher base cluster now exposes start_at, end_at, and cost_per_hour — useful for time-boxed runs and pricing-aware cluster selection on SkyPilot clouds.

:::python
from oumi import launcher

cluster, job = launcher.up(job_config, cluster_name="train-run")
print(cluster.start_at, cluster.end_at, cluster.cost_per_hour)

New & updated model configs

• Qwen3.5 0.8B — full-finetune, LoRA, and inference (HF + vLLM) recipes (#2285, [#2305]).
• Qwen3-VL 2B / 4B / 8B / 30B-A3B vision-language configs and MoE support (#2286, [#2288], [#2102]).
• GPT-OSS 120B LoRA multi-GPU training config (#2186).
• Qwen3 MoE (235B, 30B-A3B, 80B-A3B-Instruct) LoRA training configs (#2211).

• Llama 4 Scout LoRA config refresh (#2184).

  :::bash oumi train -c configs/recipes/qwen3_5/sft/0.8b_lora/train.yaml oumi infer -c configs/recipes/qwen3_5/inference/0.8b_vllm_infer.yaml oumi train -c configs/recipes/gpt_oss/sft/120b_lora_multi_gpu_train.yaml oumi train -c configs/recipes/qwen3/sft/235b_lora/train.yaml

Tool-use training: correct tool-result masking

DataCollatorForCompletionOnlyLM was extended to correctly mask tool-result tokens during completion-only training, so SFT on tool-use traces actually trains the assistant turn and not the (already-known) tool output (#2369).

---

Dependency upgrades

The big one: Transformers v5. Several small breakages were absorbed in this release so users don't have to.

• Transformers → >=4.57,<5.7 (#2317, [#2344], [#2394])
• TRL → >=0.24,<1.4 (#2298, [#2362], [#2403])
• vLLM → >=0.14,<0.21, including 0.12 support (#2275, [#2295], [#2352], [#2402])
• veRL → >=0.5,<0.8 (#2297, [#2316], [#2322])
• PEFT → >=0.17,<0.20 (#2379)
• Pydantic → >=2.11,<2.14 (#2381)
• SkyPilot → >=0.11.1,<0.13 (#1675e8e6, [#2175], [#2191], [#2349])
• datasets, torchvision, torch, wandb, liger-kernel, kernels, typer, uvicorn, pillow, responses, nvidia-ml-py all bumped (see commit list).
• TFv5 fallout absorbed: KTO/GKD import paths updated, DPO preprocessing updated, warmup_ratio deprecation handled, VL processor message-to-dict transform, tokenizer/test fixes (#2266–#2279, [#2287]–#2290).

---

New Contributors

• @idoudali made their first contribution in https://github.com/oumi-ai/oumi/pull/2186
• @AnandVishesh1301 made their first contribution in https://github.com/oumi-ai/oumi/pull/2225

Full Changelog: https://github.com/oumi-ai/oumi/compare/v0.7...v0.8