ktwu01/benchmark-radar

Hi Koutian, or probably I should say howdy, Thanks for reaching out, I took a closer look and ran a small isolated test. A few suggestions came to mind from using it: Search could benefit from semantic retrieval or stronger alias expansion. For example, exact searches for names such as LongMemEval and MemoryAgentBench did not always produce catalog records, while broader queries sometimes returned noisy matches. It may be useful to distinguish validated catalog entries from newer radar leads more prominently, with verification status, confidence, and freshness indicators. Per source health could be surfaced more clearly. In my test, the overall health check passed even though some upstream collectors had experienced timeouts or rate limits. Reproducibility metadata would be particularly valuable for benchmark research, including dataset location, license, code availability, evaluation setting, and task taxonomy. Overall, I think the project has real potential. Congratulations on open s

Goal This issue is reserved for @pengqianhan. It is independent of #303 and #445; no more work on either is required. Contribute one substantial improvement to the next Benchmark Radar technical report. Your accepted contribution must appear in the report source, supporting evidence or analysis, and the rebuilt next-draft PDF. Start here: find the report and its source Read the published reference report: Benchmark Radar v0.9.0 on Zenodo. Edit the report here: docs/technical-report/latex/main.tex. The report is a LaTeX manuscript and this file is its single source of truth. Edit it directly; the Python builder that used to hold the prose has been removed. Citations go in references.bib. Check the build notes and audited inputs: docs/technical-report/README.md. The published v0.9.0 PDF is frozen. Use it as a reference, but do not overwrite output/pdf/benchmark-radar-technical-report-v0.9.0.pdf. First: propose the contribution Choose one: an original evidence-backed analysis or result

Report target / 报告目标 Audit report finding 6.2, “Several bounded metrics are near their ceiling,” and determine which saturation claims remain valid after protocol controls. / 审计报告 6.2“若干有界指标接近上限”，判断加入协议控制后哪些饱和结论仍成立。 Research question / 研究问题 Which benchmarks truly have little measurable headroom, and which apparent ceilings are artifacts of protocol, evaluator, budget, attempts, version, or selected score track? / 哪些 benchmark 确实剩余空间很小？哪些表面饱和只是协议、评判器、预算、尝试次数、版本或分数轨迹选择造成的？ Required deliverables / 必需产物 Rebuild the curated score archive and reproduce every benchmark named in section 6.2. Predeclare the ceiling/headroom definition and protocol-compatibility rules. Audit instrument, split, metric, direction, evaluator, tools, attempts, reasoning budget, benchmark version, and registry citation for ea

Report target / 报告目标 Audit report finding 6.1, “Vendor attention has converged on a small reporting core,” before that claim is carried into the next release. / 在下一版继续使用“厂商关注集中在少数核心 benchmark”这一 6.1 结论前，对它进行审计。 Research question / 研究问题 Is the apparent reporting core robust to organization counting, benchmark identity, report selection, time window, and benchmark-family grouping? / 该核心集合在机构计数、benchmark 身份、报告选择、时间窗口和 benchmark family 分组变化下是否稳健？ Required deliverables / 必需产物 Rebuild the curated measurement layer and reproduce the current organization-by-benchmark table. Predeclare alternative organization, document, time-window, and identity/grouping rules. Test sensitivity to aliases, benchmark versions, same-family variants, missing model reports, and minimum-organization thresholds. Publish code and machine-readable tables l

Report target / 报告目标 Add an evidence-backed study to report section 6 on agent progress, recurring weaknesses, and benchmark measurement gaps. / 为报告第 6 节增加基于证据的研究：Agent 的进展、反复出现的弱点和 benchmark 测量盲区。 Research question / 研究问题 Which agent capabilities or failure modes recur across benchmarks, what evidence demonstrates them, and which weaknesses are only implied by benchmark design? / 哪些 Agent 能力或失败模式在多个 benchmark 中反复出现？哪些有实证支持，哪些只是由任务设计暗示？ Required deliverables / 必需产物 Predeclare the agent scope, benchmark inclusion rule, time window, and evidence threshold. Use reproducible Radar searches and cited benchmark artifacts to build a reviewable capability/failure taxonomy. Publish a coding guide and machine-readable coded table with benchmark IDs, evidence excerpts or locations, dates, and citations. Separate demonstrated model fai