vllm-project/aibrix

Parent issue: #2989 Background AIBrix documents vLLM-Omni as exposing OpenAI-compatible multimodal APIs, including /v1/audio/speech. The Gateway currently supports audio transcription and translation but not speech generation. Speech generation has a different transport contract: a JSON request, a binary audio response, potentially multiple streamed response chunks, no JSON usage or SSE processing, and no async job registration. Proposed behavior Support POST /v1/audio/speech. The Gateway should: parse the JSON request; require a non-empty model; require a non-empty textual input; preserve all other request fields; use existing non-PD model routing; stream successful binary responses without parsing or mutation; preserve upstream audio headers. Validation Add a focused validator such as: func validateAudioSpeechRequest( requestID string, requestBody []byte, ) (model, message string, errRes *extProcPb.ProcessingResponse) Validate only a JSON object root, a non-empty string mod

Parent issue: #2989 Background SGLang v0.5.21 adds and extends POST /v1/score for pointwise and setwise scoring. References: https://github.com/sgl-project/sglang/releases/tag/v0.5.21 sgl-project/sglang#38965 AIBrix Gateway supports related endpoints but does not currently register or parse /v1/score. Proposed behavior The Gateway should: accept POST /v1/score; parse its JSON request; require a non-empty model; require a non-empty items field; preserve all request fields; use existing non-PD routing; process token usage when present; preserve the upstream score response without transforming score matrices. AIBrix must require model even when SGLang can use a server-side default, because the Gateway needs it for backend selection. Validation Add a focused validator such as: func validateScoreRequest( requestID string, requestBody []byte, ) (model, message string, errRes *extProcPb.ProcessingResponse) Validate only Gateway requirements: JSON object root; non-empty string mod

Parent issue: #2989 Problem The Gateway validates whether a routing strategy is recognized, but not whether it is compatible with the request endpoint. The PD router mutates generation-specific fields such as max_tokens, max_completion_tokens, stream, stream_options, and min_tokens. Applying this transformation to embedding, ranking, scoring, multimodal, or engine utility endpoints can create invalid requests or undefined behavior. Supported PD endpoints Initially permit PD only for: /v1/chat/completions /v1/completions Reject PD for every other endpoint, including Responses, Messages, Embeddings, Rerank, Classify, Decisions, SystemOne, Score, image/video/audio APIs, Tokenize, and Pooling. Responses and Messages may be enabled later after dedicated PD compatibility tests exist. Proposed behavior Add a small focused compatibility check, for example: func validateRoutingStrategyForPath( requestPath string, strategy types.RoutingAlgorithm, ) error A switch or allowlist is suffic

Parent issue: #2989 Problem Several Gateway validators log the complete request body when JSON parsing fails. Other request-processing logs include parsed prompts or messages at V(4). These diagnostics are useful, but request bodies can contain user prompts, tool arguments, conversation history, or large encoded data. Request contents should not appear at normal production log levels. Proposed behavior At normal log levels, parsing failures should record only: request ID; endpoint; sanitized parse reason; request body size. Raw JSON content may be logged only at klog.V(6). Verbose logging must: include at most 4 KiB of the request body; record whether the value was truncated; never log multipart file contents, filenames, or form values; apply the same policy to parsed prompt or message fields; require no new environment variable. Example normal log: klog.ErrorS(nil, "error to unmarshal request", "requestID", requestID, "endpoint", requestPath, "reason", jsonParseReason(

Parent issue: #2989 Problem The Gateway request parser and ModelRouter already support: /v1/messages /v1/audio/transcriptions /v1/audio/translations However, these paths are missing from the deployed shared reserved-router in config/gateway/gateway-plugin/gateway-plugin.yaml. A request must first match this route to enter the external processing service. The plugin then extracts the model from the request body, writes the internal routing headers, clears the route cache, and rematches the request onto a model route. Without the first route, the existing parser and ModelRouter support do not form a complete deployment path. Proposed change Add these existing endpoints to the shared reserved route: - path: type: PathPrefix value: /v1/messages - path: type: PathPrefix value: /v1/audio/transcriptions - path: type: PathPrefix value: /v1/audio/translations Verify the rendered Kustomize output as well. Testing Add regression tests confirming that: all three paths ar