ShenSeanChen/waku-agent

Found by @savvassavvidis-eng while reporting #132, and confirmed live. GET /v1/models still returns ids OpenAI has deprecated: gpt-5.3-chat-latest STILL LISTED -> 404 on use gpt-5.2-chat-latest STILL LISTED -> 404 on use gpt-5.1-chat-latest STILL LISTED -> 404 on use gpt-5-chat-latest STILL LISTED -> 404 on use waku/ops/catalog.py builds the Settings model picker straight from that response, so the dashboard presents four models that cannot answer a turn. Pick one and every message dies on NotFoundError, with nothing in the UI suggesting the model was the problem. #132 fixed the default. This is the other half: a user who opens the picker can still walk into it. The catch There is no deprecated field on the catalog response — the ids simply keep appearing. So this cannot be solved by reading a flag. Options, roughly in order of how much they cost: Deny-list the known-dead ids in catalog.py. Cheap, works today, and goes stale the next time OpenAI retires something.

After #81 landed, evals/judge/ has two files that configure GEval differently: test_response_quality.py uses criteria — DeepEval generates its own rubric from a prose description test_retrieval_gate_accuracy.py uses evaluation_steps — the steps are written out explicitly Two conventions in one directory is how the next person writing a judge eval guesses wrong. Which way to converge, and why Toward evaluation_steps. From #81, with a worked example: given only criteria, the generated rubric kept contradicting the stated rules between runs — in one case scoring a search query as though it were an answer ("it does not actually answer what the user had for breakfast"), a requirement nobody asked for. Removing the generation step removes that drift; the rules cannot be paraphrased away run to run. Reproducibility is the whole point of a scored eval. A rubric that is regenerated per run is a moving yardstick. Scope Rewrite test_response_quality.py's criteria as explicit evaluation_s