pegainfer-project/pegainfer

Problem Gemma 4 still launches on the legacy engine contract (LaunchedEngine::Handle + per-request TokenEvent), while Qwen3 and pegainfer-sim already speak the step contract (LaunchedEngine::Stepped). The two generations currently coexist in pegainfer-frontend (bridge.rs vs bridge/stepped.rs). Every remaining Handle line keeps that dual stack alive. Gemma 4 is a good next cut: single GPU, one scheduler, no LoRA / TP / EP / P/D, and prefix cache is resolved inside the engine rather than via frontend KvPrefix. Today: pegainfer-gemma4/src/model_line.rs returns LaunchedEngine::Handle. pegainfer-gemma4/src/engine.rs owns its own thread, blocking_recvs when idle, and emits TokenEvent from admit / mixed gather / async-prefill join / chunked walk / decode. Cancellation is TokenSink send-failure / channel close, not RequestControl::abort. There is no Scheduler::load() occupancy feed (running / waiting / KV pages). Related: Gemma 4 roadmap #758. Contract source of truth: docs/subsystems/fronte

Problem Gemma 4's checkpoint-backed gates are valuable but have no continuous compile contract or checked-in execution manifest. At 69c4e149: pegainfer-gemma4 has 19 ignored library gates: 8 in src/engine.rs and 11 in src/serve_oracle.rs. The gates cover HF numeric parity, long/window context, generation, mixed admission, async prefill, prefix cache, CUDA Graph replay, cancellation, and lifecycle behavior. GitHub CI does not build or Clippy-check pegainfer-gemma4 --features gemma4 --all-targets. No checked-in runner names the maintained gate set. Engine gates document --test-threads=1, but a generic --ignored command does not encode the suite membership or prove that every expected gate still exists. The tests currently fail closed when PEGAINFER_TEST_MODEL_PATH is absent and pin their checkpoint/fixtures. The decay risk is that feature-only test code can stop compiling or individual gates can fall out of the informal command set. Related model roadmap: #758. Direction Add feature-on

Problem The HD256 and HD512 QK/RoPE smoke suites maintain a second Rust implementation of substantial GPU semantics. At 69c4e149: pegainfer-kernels/tests/hd256_qk_rope_plain_smoke.rs is 783 lines and has four closed-form tests covering RMSNorm, RoPE pairing/tail behavior, paged-KV address calculation, and pool writes. pegainfer-kernels/tests/hd512_qk_rope_smoke.rs is 951 lines and has three analogous closed-form tests. Both files independently define helpers such as inv_rms, normed, expected_prep, expected_full, expected_pool, and assert_pool. These tests have value, but seven large host-side reconstructions create a parallel implementation that must evolve with the CUDA contract. A shared specification mistake can also be repeated in both the kernel and its local oracle. The newer row-offset tests are a better shape: compare the offset GPU arm against the same production GPU path at zero offset, then verify untouched prefix/suffix/KV sentinels. They should remain. Direction Keep a s

Summary Under the PerToken batch-invariance policy, each bucket-N decode CUDA graph bakes ~36 layers × 6 projections × N cublasGemmEx nodes, because the PerToken GEMM oracle (gemm_per_token_cuda, openinfer-kernels/csrc/shared/linear.cu:772) loops one GEMM per row. Graph-exec memory therefore grows linearly with the bucket size; across all 36 decode buckets this sums to ~1.8 GB for Qwen3-4B (measured on RTX 5070 Ti 16GB via cuMemGetInfo probes during a full bucket sweep: drain starts around bucket 56, ~-134 MB by bucket 224, OOM near bucket 232 with util=0.90). The KV-cache memory budget (util, default 0.90) never reserves this per-policy graph memory, so PerToken + CUDA graph can OOM — deterministically at startup once decode graphs are pre-captured, or mid-serving when capture is lazy. What is unaffected Tuned (production default) and Pin policies: their graphs have O(layers) nodes, measured flat at ~2 MB/bucket across the sweep. Eager (non-graph) decode. Proposed directions Bu

Context OpenInfer pins FlashInfer v0.6.14 at 19f1a41e6b21f0c422d775e377b6fdf9a1fc9d23. Qwen3.5 prefill currently runs a seven-stage Triton-AOT Gated Delta Rule pipeline generated from openinfer-kernels/tools/triton/gated_delta_rule_chunkwise_kernels.py. This is the Qwen3.5 model line's only build-time Triton dependency. The pinned FlashInfer tree now has a fused chunk_gated_delta_rule prefill backend for the architectures relevant to current Qwen3.5 work: SM100: flashinfer#3001 SM90: flashinfer#3477 SM120: flashinfer#3479 Follow-up prefill work: flashinfer#3742 FlashInfer's own benchmark matrix includes Qwen3.5 4B/9B (Hq=Hk=16, Hv=32), 27B (Hv=48), and larger-family head configurations. Its tests cover the 16/16/32 GVA shape used by 4B and 9B. This makes an upstream-backed migration plausible. It is not a drop-in wrapper. Integration constraints 1. The pinned implementation is CuTe DSL, not a reusable C++ launcher All pinned SM90/SM100/SM120 prefill backends live under flashinfer/gdn