novitalabs/pegaflow

Problem python/README.md lines 37-52 show engine.put() / engine.get() / engine.remove() API examples that do not exist in the current codebase. PegaEngine actual API is register_context_layer() / batch_load_kv_blocks(). Users don't directly call the Python binding API — the intended usage path is through vLLM: vllm serve <model> --kv-connector PegaKVConnector ... Fix Delete the "Basic KV Storage" section entirely Keep the vLLM usage section and SGLang examples (mark SGLang as experimental/unsupported) Ensure the README serves as a clear landing page for PyPI visitors Reference Tracked in docs/projects/python-binding.md §3.4

Summary pegaflow-core/src/uring.rs currently lacks an IO-focused benchmark path that reflects real SSD behavior. Criterion-style microbenchmarks are less suitable for this workload. Proposal Add a custom Rust benchmark for uring with [[bench]] + harness = false and a fixed, reproducible workload. Key points: Use std::time::Instant for timing Measure mixed read/write workloads with configurable: total bytes concurrency block size io_depth / shard count Report: throughput (MB/s) IOPS latency percentiles (p50/p95/p99) via hdrhistogram Why Better control over IO patterns than Criterion microbench loops Closer to production access patterns for SSD cache traffic Easier to compare and align results with fio baselines Alignment with fio Please document one or two reference fio profiles (same block size, queue depth, read/write ratio) so benchmark outputs can be compared apples-to-apples. Acceptance Criteria New custom benchmark compiles and runs locally Outputs MB/s, IOPS, p50/p95/p

Summary Currently we have two allocation strategies in AllocStrategy: WriteCombined: Uses cudaHostAllocWriteCombined, optimized for CPU→GPU transfers but bypasses CPU cache HugePages: Uses mmap(MAP_HUGETLB) + cudaHostRegister We should add a third Regular (or Default/Cached) strategy that uses cudaHostAllocDefault for bidirectional workloads where CPU cache matters. Motivation Write-combined memory performs poorly for CPU reads (no cache). For KV cache offloading with bidirectional transfers (GPU↔CPU), regular pinned memory may be more suitable. Tasks Add AllocStrategy::Regular variant in pinned_mem.rs Update CLI: change --use-hugepages bool flag to something like --memory-mode with choices: regular, write-combined, hugepages Propagate the strategy choice through PinnedMemoryPool → PinnedAllocator → PegaEngine Update tests and documentation Code References pegaflow-core/src/pinned_mem.rs: AllocStrategy enum and PinnedMemory::allocate pegaflow-core/src/pinned_pool.rs: Pinne

Summary Create an end-to-end accuracy test script (similar to bench_kv.py) that verifies Pega's store and load functionality works correctly by running the same requests twice and comparing outputs. Motivation We need to ensure that Pega correctly stores and loads KV cache without affecting model output accuracy. Running the same requests twice should produce identical results - the first run stores the KV cache, and the second run loads from cache. Any discrepancy would indicate a bug in the store/load logic. Proposed Implementation Create a new script tests/e2e/test_accuracy.py (or bench_accuracy.py) that: Starts a backend server (vLLM or SGLang) with Pega enabled Sends a set of deterministic requests (temperature=0) to the server Records the outputs from the first run (KV cache store) Sends the same requests again (KV cache load) Compares outputs between the two runs Reports any mismatches Requirements Support both vLLM and SGLang backends Use deterministic generation settings (t

Introduce a new parameter prefetch_threshold to filter out low-hit entries. Only items exceeding this threshold will be prefetched, helping to reduce noise from infrequent hits and optimize prefetching efficiency.