#110 Provider setup should probe streaming native-tool compatibility

[Anonymous]

Originally created by: yablokolabs

Summary

A successful /v1/models request proves that an OpenAI-compatible endpoint is reachable, but not that the selected route supports Atomic's actual contract: streaming chat completions, supplied tools, native tool calls, and the configured model.

During Yabloko Labs' evaluation, OmniRoute itself was reachable, but tested routes failed with STREAM_EARLY_EOF, insufficient_quota, unsupported-model errors, or HTTP 400 only when tools were supplied. These were primarily route/provider limitations, not proof of an Atomic core defect. Atomic can still make diagnosis much faster.

Current behavior

The OpenAI provider health check only requests /v1/models:

• openai-provider.ts (github.com)

The provider wizard saves and activates the provider, but does not exercise a minimal streaming tool call before reporting success:

• providers-orchestrator.ts (github.com)

Proposed behavior

Add an explicit provider-contract test in the provider UI/CLI.

The primary conformance probe should send a synthetic, side-effect-free function definition using a probe-specific forced named-tool choice (or provider-equivalent required mode), then inspect the streamed response for one complete native tool call. The synthetic call is diagnostic data only: it must never be dispatched to Atomic's tool registry.

If a provider does not support forced tool choice, an optional tool_choice: auto probe may still run, but a normal text response with no tool call is inconclusive, not proof that tools are unsupported.

The result should distinguish at least:

• endpoint/authentication failure;
• quota or provider-routing failure;
• unknown/unsupported model;
• forced/required tool choice rejected by the route;
• tools payload rejected by the route;
• streamed response ended early;
• malformed/incomplete SSE tool-call deltas;
• no tool call under auto — inconclusive.

Acceptance criteria

• Users can run a side-effect-free contract probe before starting an agent turn.
• The conformance probe uses the configured model, streaming transport, tools payload, and forced named-tool or required mode when supported.
• A synthetic probe call is inspected only and is never executed or registered as a real Atomic tool.
• A text response under tool_choice: auto is reported as inconclusive, not incompatible.
• Failures are categorized with actionable messages rather than only HTTP N.
• Credentials and full provider response bodies are redacted from UI events and logs.
• A failed probe does not silently mark the provider as fully compatible.
• Custom endpoints can still be saved with an explicit warning/override because some providers block synthetic probes.
• No probe runs on every turn; it is setup-time or explicitly requested.
• Tests cover endpoint success with no tools, forced-tool rejection, auto-mode text response, early EOF, quota failure, malformed deltas, and a valid streamed tool call.