swellweb/reame

Our numbers come mostly from a 4-core Oracle ARM instance. New Oracle Always-Free accounts get 2 OCPU / 12 GB, so most people arriving today will see something different from our table — and a benchmark table that does not match the reader's machine is worse than no table. What to do: run on a 2-core free-tier instance (Oracle, or any 2-core VPS) and report: decode tok/s for a model of your choice (reame run <model> picks defaults automatically) the fact exam from bench/ RAM used at rest and under load Post the raw output and we will add the row to docs/BENCHMARKS.md with credit. Zero code required — this is measuring, and it is the single most useful thing an outsider can contribute right now.

Reame runs on a free-tier ARM box today, but getting there is a manual build. A Dockerfile (multi-arch: linux/arm64 + linux/amd64) would make "try it on your own box" a one-liner. What to do: Multi-stage build: compile with the toolchain, ship only the binary and its runtime deps. Sensible defaults for a 2-core / 12GB free-tier instance, mirroring config/reame.conf. Volume for the model directory and one for the KV cache (the cache is the whole point — it must survive container restarts). A docker-compose.yml example with the ARCA daemon alongside the server would be a bonus. Constraint that matters: the image should stay small enough to be practical on a free-tier box with limited disk. Please post the final image size in the PR.

The fact-sheet extractor in the compression pipeline currently knows Italian business pages: prices per m², opening hours, VAT numbers, warranties, service conditions. That is the domain we measured, and it is the reason the numbers are what they are. What to do: propose (or implement) extraction rules for a different document family you actually work with — invoices, support tickets, CVs, product specs, medical reports, whatever. The two rules that must hold, because they are what makes this safe: Values are copied byte-for-byte from the source. Never paraphrase a number, a name or a date. A condition travels with the value it qualifies. "9:00–12:00" without "by appointment only" is a wrong answer waiting to happen — we have the failing test to prove it. Even without code, a comment describing the patterns in your domain and 2–3 real (anonymised) examples is genuinely useful. Context on why formatting matters this much: docs/BENCHMARKS.md.

The exam scorer currently prints a human-readable table. For CI runs and for comparing results across machines it should also be able to emit machine-readable output. What to do: add --json to bench/run_fact_exam.py so it prints one JSON object per input form: form name, token count, TTFT, score, critical score, hallucination and miss counts, plus model name and a timestamp. Keep the existing text output as the default — the point is to add a mode, not replace one. Why it matters: we want people to post reproductions in #7, and a JSON blob is much easier to compare than a screenshot. Small, self-contained, pure Python. Good first change if you want to get familiar with the repo.

Everything behind our headline claim is in bench/: one customer page in three input forms, 20 adversarial questions, and a scorer. python3 bench/run_fact_exam.py --server http://127.0.0.1:8080 --model <your-model> What would help: run it against whatever you already have — a different model, a different CPU, an x86 box, a Raspberry Pi, even a GPU — and post the output here with your hardware and model. We are especially interested in results that contradict ours. The claim is that reformatting the input as label: value makes small models both faster and more accurate; if that does not hold on your setup, that is a finding worth publishing and we will put it in the benchmarks. No C++ needed — just Python 3 and a running OpenAI-compatible server.