vllm-project/llm-compressor

Background LLM Compressor utilizes a tracing system to trace through a model and break the model into subgraphs for sequential processing. At the end of tracing, the model is represented as a list of Subgraphs, with each subgraph having a submodules method for getting all the modules called by that subgraph. Ideally, subgraphs have the following property: set(subgraph.submodules() for subgraph in subgraphs) == set(model.modules()) - sequential_ancestors Where sequential_ancestors are any modules which (indirectly or directly) call the sequential_targets. However, in most models this is not the case, as layers such as the embedding layers are called when tracing the model, but do not show up in any of the subgraph.submodules(). This should be impossible, as any time that a module is called during tracing which is not an ancestor, it should appear as a call_module op in the subgraph (source). Proposed Work Determine why the tracing system fails to include these modules in the subgraph

Background Model Free PTQ allows users to specify how many workers to use when processing checkpoints. However, this decision should essentially be automated, since we already know how much memory each job will take as well as how much device memory is available and how many safetensors files need processing. Proposed Changes Implement an "auto" option which selects a good value given the current hardware and task. The changes should be mostly in compressed-tensors/src/compressed_tensors/entrypoints/convert/convert_checkpoint.py::convert_checkpoint and be fairly minimal. If available, please test using a multi gpu setup or ping me to run added tests with multigpu.

Background Many models use 3d tensors to represent moe weights. LLM Compressor currently handles these by first separating them into 2d linear experts, then saving as 2d. However, vLLM often does not support loading 2d weights natively, which requires extra effort to support in vLLM. Current flows for 3d checkpoints involve saving the compressed model in 2d, then using a custom script to repack them into 3d after the fact. See: RedHatAI/Inkling-NVFP4-FP8-BLOCK model.llm.layers.32.mlp.experts.w2_weight -> model.llm.layers.19.mlp.experts.w2_weight.input_global_scale -> model.llm.layers.19.mlp.experts.w2_weight.weight_global_scale -> model.llm.layers.19.mlp.experts.w2_weight.weight_packed Current LLM Compressor utilities such as repack_moe do not support repacking quantized weights and not pack into a format which can be picked up by conversion_mappings. Suggested Changes Modify repack_moe to handle comp

Background LLM Compressor requires information as to which transformers modules are fused by vLLM. These "fused mappings" are used to Ensure that nvfp4 global scales are shared between fused modules Ensure that fused modules have the same quantization This check is implemented in HIGGS and model_free_ptq This check needs to be implemented in oneshot (followup) However, there are currently multiple sources of truth, and these sources of truth are out-of-date with respect to recent model architectures and attention implementations. Requested Changes Consolidate the fusion mappings below into a single mapping in a single file llm-compressor/src/llmcompressor/observers/helpers.py Lines 162 to 215 in 3ae3cb1 FUSED_LAYER_NAMES = [ # MLP / expert layers have fused gate_up_proj ("gate_proj", "up_proj"),

Background LLM Compressor and Compressed Tensors both use logic which flattens and unflattens nested structures with tensors in them. The two examples which come to mind are below. I believe these are the only examples IntermediatesCache send_tensors torch.utils._pytree does a very similar thing, it may be possible to leverage this module to perform these operations in a more robust way. Suggested Steps Find all examples of nested flattening/unflattening of tensor structures Try using torch.utils._pytree to do the flattening/unflattening Add additional tests to determine edge case behavior