mingti-org/phyai

Summary Add support for Marlin INT4 kernels and AWQ/GPTQ quantized linear layers in phyai. Motivation Marlin provides efficient FP16xINT4 matrix multiplication kernels for LLM inference. Supporting Marlin with AWQ/GPTQ checkpoints would allow phyai to run common 4-bit quantized models with better memory efficiency and throughput on supported NVIDIA GPUs. References https://github.com/IST-DASLab/marlin

Summary Add support for running the PyTorch backend of phyai on Apple Silicon GPUs through PyTorch's MPS backend (torch.device("mps")). Motivation PyTorch supports GPU acceleration on macOS through the Metal Performance Shaders (MPS) backend. Supporting MPS in phyai would allow users to run development, inference, and lightweight training workflows locally on Apple Silicon Macs without requiring CUDA hardware. This should provide a native PyTorch-based Apple Silicon path, separate from MLX support. References https://docs.pytorch.org/docs/stable/notes/mps.html https://developer.apple.com/metal/pytorch/

Summary Add support for MXFP4-quantized Linear layers in phyai. Motivation MXFP4 is an OCP microscaling FP4 format that stores 4-bit floating-point values with block-wise shared scaling. It can significantly reduce memory footprint for large model inference while preserving better numerical behavior than simple INT4-style quantization. Supporting MXFP4 linear layers would let phyai run models or checkpoints that are distributed with MXFP4 weights, and provide a hardware-agnostic FP4 path where runtime support is available. References https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf https://huggingface.co/docs/transformers/en/quantization/mxfp4

Summary Add support for NVFP4-quantized Linear layers in phyai. Motivation NVFP4 is NVIDIA’s FP4 format for Blackwell Tensor Cores, targeting efficient low-precision inference with 4-bit values, 16-element block scaling, and FP8 scale factors. Since LLM inference is heavily dominated by Linear / GEMM workloads, supporting NVFP4 linear layers can reduce memory bandwidth and improve throughput on compatible NVIDIA Blackwell hardware. References https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ https://docs.pytorch.org/TensorRT/user_guide/shapes_precision/quantization.html https://docs.pytorch.org/ao/stable/api_reference/generated/torchao.prototype.mx_formats.NVFP4DynamicActivationNVFP4WeightConfig.html

Summary Add backend support for SandAI-org/MagiAttention, a distributed/context-parallel attention implementation designed for ultra-long-context training and heterogeneous attention masks. Motivation MagiAttention provides a scalable distributed attention backend with support for flexible mask patterns, computation load balancing, zero-redundant communication, and computation/communication overlap. Supporting it in phyai would make it easier to run long-context training workloads that require context parallelism and efficient distributed attention. References https://github.com/SandAI-org/MagiAttention