vllm-project/semantic-router

Problem The model runtime's cuda accelerator is implemented and unit-tested, but not validated on NVIDIA GPUs. Its fused kernels stay disabled until a bit-exactness record exists, and the user guide lists NVIDIA GPUs as "work but not yet validated". Scope On NVIDIA GPUs (for example H100, A100 or L4): A GPU smoke of the CUDA router image as the charts run it. Golden answers for every built-in model the CUDA image serves on the GPU. Parity records per family against the released references: Decision 1.0 and Decision 2.0 panels, Vela 2.0's request panel, and the Vela 1.0 path. A bit-exactness record for each fused kernel slot before it is enabled on CUDA, plus pinned FLA kernel choices for the Qwen3.5 decoders. Performance records against the legacy path or the released runtime, to the repository's no-regression standard. The user guide and design doc updated from "unvalidated" to "validated" for what was measured. Acceptance All records committed under src/model-runtime/docs/records/,

Goal Make it clear what each router checkpoint was trained on. Eight Vela cards (Domain, Guard, PII, Safety, Hazard, FactCheck, Feedback, Modality) don't name their training data, either in the datasets metadata or in the text, and Halu lists its two LettuceDetect datasets only in the metadata. The Vela Shield card already does this: every source with licence, revision and rows used, and an exposure column in each eval table saying which compared model trained on the same source. It matters when choosing eval sets. The older intent classifier names MMLU-Pro as its training data and scores 0.944 on the 12,032-question MMLU-Pro test split, against 0.496 on 1,988 MMLU-CF questions and 0.606 on 1,882 SuperGPQA questions mapped to the same labels. Vela Domain scores 0.709, 0.529 and 0.587. With the older card a reader can see why its first number is high. The Vela card names nothing, and its MMLU-Pro number is partly on its own training data too (#4300). Proposed Workgroup Router Models & I

Goal Part of #4241. SR Bench 2.0 should show how task success and continuity hold up when a provider fails partway through a task. Today the provider mocker's behavior is set once per process through environment variables, including one global response delay (settings.py). bench/openai_fault_proxy.py, from #1989, injects faults for the soak harness that @drivebyer built in #3264 and @xuuuge extended to streaming in #3447. Its faults also apply to the whole process. Neither lets one benchmark task ask for "return 503 on this task's third call" while other tasks run normally. @ZireaelK's #3428 adds stall probes to the mocker for timeout tests, which a schedule could reuse. This builds on that work, so @drivebyer, @xuuuge or @ZireaelK get first pick. Proposed Workgroup Evaluation & Quality Scope Let the provider mocker read a fault schedule keyed by a request-scoped value, where each entry picks a call index and a fault: an HTTP status, a delay, or a stream cut short. Let an sr-bench man

Proposed Workgroup Developer Experience & Ecosystem Describe the enhancement While checking whether a model is live on Build → Config → Models (/config/models), the Live verification control causes neighboring table columns to shift. The Pricing column’s $ symbol appears appear and disappear or slide horizontally during the Checking state, even though pricing data does not change. Screen recording: Screen.Recording.2026-09-11.at.4.50.28.PM.mov What happens Clicking Check runs a real inference probe (POST /api/models/verify). The control (ConfigPageModelLiveVerification) lives in the Live column, immediately before Pricing in ConfigPageModelInventoryPanel.tsx. Phase Status label Button label Extra UI idle Not checked Check — pending Checking Checking… yellow pulsing dot verified Live Check again green dot failed Unavailable Check again inline error message Each phase changes the width of the Live cell: Variable button label

Goal Remote classifier backends (http_classify and other connector-backed classifier calls) should stop issuing live RPCs for a bounded cooldown after consecutive timeouts or unavailability, instead of waiting on a dead endpoint on every request. The breaker only decides whether to place the call. A skipped call must still surface as the existing unresolved/timeout error so decision-tree Unknown, rules.on_unknown, and prompt-guard on_error keep their current meaning. This is the cooldown follow-up called out on #3106 and in #2759, not a replacement for those policies. Proposed Workgroup Router Models & Inference Runtime Scope Per remote-classifier backend instance in a process: consecutive-failure threshold, open interval, and a single half-open probe. On open, skip the RPC and return the same typed timeout/unavailable error the connector already returns. Default off, or on only for remote HTTP/MCP-style backends. In-process local models (mmbert / Candle) stay unchanged. Jailbreak/PII