higress-group/higress

Why do you need it? OpenAI Responses API contains protocol-level state that is tied to the provider or deployment that produced the previous response. The important fields are: previous_response_id reasoning item ids encrypted_content When AI Route load balances a follow-up request to a different provider or deployment, the upstream provider cannot resolve state created by another backend. This breaks multi-turn Responses API workloads under multi-provider or multi-deployment routing. The problem is visible with Azure OpenAI multi-deployment routing: a first request can succeed, while a follow-up request with previous_response_id fails when routed to another deployment. How could it be? Add optional Responses API affinity support in ai-proxy. Recommended configuration: responsesAffinity: enabled: true encodeResponseId: true encodeItemId: true encodeEncryptedContent: true providerUnavailablePolicy: failFast Request processing: Detect Responses API requests. Decode provider

Why do you need it? When an AI Route is configured with multiple AI providers or multiple deployments of the same provider, some workloads need provider-level session affinity. Today, AI Route can distribute traffic across multiple upstream AI providers, but there is no generic way to pin a logical session to the same provider instance. This is important for LLM gateway scenarios where the selected provider/deployment may hold provider-side state, cache, or session context. Without provider-level affinity, consecutive requests from the same logical session may be routed to different providers, causing inconsistent behavior or request failures. The session key should not be limited to HTTP headers only. In LLM protocols, the session key may come from different places, for example: Request header Cookie JSON body field Dynamic metadata set by an earlier plugin A concrete example is a request body field such as: { "model": "gpt-4o", "callOptions": { "stickySessionId": "session-1

Higress版本 使用2.1.5Higress，拉取主分支代码ai-load-balancer在本地打包wasm 基于redis的最小请求 看代码只能处理瞬时的负载均衡，一段时间内持续请求就退化为随机数了，长时间请求负载均衡效果非常差，还不如默认的轮询好用。 底层代码类似加锁，可以考虑加个滑动窗口的？ 检测模型可用 https://higress.ai/blog/higress-gvr7dx_awbbpb_qg6l3tk0zc4dughn#%E5%89%8D%E7%BC%80%E5%8C%B9%E9%85%8D%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1-1 技术文章写了：在全局最小请求数负载均衡中我们重点关注了请求异常（例如后端服务不可访问、客户端断连、服务端断连等）情况下的处理，通过在HttpStreamDone阶段统一进行计数的变更可以保证异常中断的请求也能够得到计数的更新，避免因请求异常导致服务计数异常情况。 但是实测kill掉一个模型pod，后续请求会持续报错。 这个问题我们配上pod健康检测就解决

Why do you need it? 目前看支持了 vLLM，想问下后续有考虑支持 SGLang、Triton TensorRT-LLM 等其它推理引擎吗？ https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions/ai-load-balancer/endpoint_metrics/backend/vllm How could it be? 目前项目同时使用 vLLM 和 SGLang，若监控指标负载均衡能同时兼容两者，会更加便捷一些。 Why do you need it? At present, vLLM is supported. I would like to ask if you will consider supporting other inference engines such as SGLang and Triton TensorRT-LLM in the future? https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions/ai-load-balancer/endpoint_metrics/backend/vllm How could it be? The current project uses both vLLM and SGLang. It would be more convenient if the monitoring indicator load balancing could be compatible with both.