roy-tong/AgentMeasure

Found twice in the 2026-10-08 audit round: when a usage row carries a model id the tool does not recognize, some cost trackers fall back to a default model's price instead of flagging the row. Worst observed case: an unknown gpt-6-astra row silently priced as a Sonnet default — a 27x undercount the user never sees (zihenghe04/CCDash#8, camelyello-88/llmstat#1). Task: add a conformance vector for this pattern under conformance/: A small synthetic export (2 rows is enough): one row with a well-known model id, one with a made-up future id (e.g. model-2099a). Expected verdicts: the known row is priced normally; the unknown row must be reported as UNPRICED/UNPROVABLE — never silently zero and never a default-model price. Follow the layout of an existing vector directory (README + fixture + expected outcomes; see conformance/evidence/ for examples). Acceptance: the vector runs under the existing harness and fails any implementation that substitutes a default price.

While auditing pricing tables on 2026-10-08 we found that OpenAI's official pricing page now lists an explicit Cache writes column (example: gpt-6-astra $10 base / $12.50 cache write = 1.25x). An earlier assumption in our audit notes — "OpenAI does not bill cache writes" — is therefore outdated. Task: check the pricing references under registry/ (and any vendored OpenAI price data) for cache-write coverage: If cache-write rates are missing, add them with the source link and the date you checked. Anchor every number to the official pricing page as of a specific date (our 口径: no undated numbers). Add or extend a fixture under conformance/ that distinguishes cache-write tokens from cache-read tokens so the two rates cannot be conflated. Acceptance: healthcheck passes with the new data; the README table (if any) and registry/vendor-rules.json agree; every added number has a source + date.

Why: our audience is CX/support-ops people who get billed per AI resolution (Fin, Zendesk AI, Agentforce). They do not need the spec — they need a 10-minute path: export last month's conversations, run the local checker, read the one-page statement (confirmed vs assumed resolutions, dollar delta). What to do: write a walkthrough doc (markdown, website/blog or docs/) for one vendor (Intercom Fin is the best-documented export). Steps with real commands and expected output shapes. Acceptance: a newcomer can follow it end-to-end without reading AMS-1; every number in the example output is reproducible; no real customer data. Good to know: settle --format md and the recount page (website/recount.html) are the two existing entry points; the blog post "resolution bill check" covers the framing — the walkthrough should be more hands-on than that.

Why: gateways that only read message_delta silently drop prompt-side usage when a provider puts input/cache buckets only in message_start. This is a real-world pattern (see our llmio#71 thread and one-api-pro#13 precedent) and we want it pinned as a conformance vector so local checkers can catch it. What to do: add a vector under conformance/vectors/ with a minimal SSE transcript pair: (a) usage in message_start + final message_delta, (b) same stream but usage only in message_delta. Expected verdicts: (a) PASS, (b) FAIL with the specific check that fires. Acceptance: both vectors run green under the conformance runner; the FAIL case documents which named check catches the missing input bucket. Good to know: look at OUT-001..005 for vector file conventions. Keep transcripts synthetic and minimal — no captured customer traffic.

Why: AMS-1 settlement statements start from vendor exports, and each vendor names the same columns differently. We already map Intercom and Zendesk native headers to canonical fields (registry/vendor-rules.json) — adding a vendor is a data-only change, no engine code needed. What to do: pick a vendor we do not cover yet (e.g. Freshdesk, Help Scout, Gorgias, or Salesforce Agentforce conversation exports), find their documented CSV column names, and add a vendor block with header aliases. Acceptance: aliases map only to existing canonical fields (do not invent new semantics) run python scripts/gen_web_rules.py and commit the regenerated website/vendor-rules.js healthcheck parity tests stay green (run the conformance suite locally) Good to know: registry/ is the single source of truth; healthcheck and website copies are regenerated. See commit ec0d06a history for the Zendesk alias example (space-form headers like "Ticket ID" / "Solved at").