lightly-ai/lightly-train

🧠 Describe the Bug Fine-tuning object detection from a COCO-pretrained checkpoint onto a different number of classes does not re-initialize the classification head. It silently keeps the checkpoint's leading rows. _reuse_or_reinit in src/lightly_train/_task_models/object_detection_components/hooks.py:168 truncates when the checkpoint has more classes: if num_classes_checkpoint > num_classes_module: # Checkpoint has more classes: truncate to module's expected size. adjusted_weights = checkpoint_weight[:num_classes_module, :].to(device) So fine-tuning COCO (80 classes) onto 3 classes starts from the COCO rows for person, bicycle and car. Training still converges, but from an arbitrary bias that depends on COCO's class ordering. This contradicts: The train_object_detection docstring: "Otherwise, the classification head will be re-initialized." The module docstring, which claims the hooks work by "truncating or zero-/re-initializing". src/lightly_train/_task_models/eomt/hooks.p

Sub-issue of #965. Description Image classification has no way to deal with imbalanced datasets. The only loss setting today is label_smoothing, and both losses are built without any class weighting: image_classification/train_model.py image_classification_multihead/train_model.py Both tasks should be covered. Expected behavior Add a class_weights argument next to the existing label_smoothing, following the general design in #965: class_weights: Literal["auto"] | dict[str, float] | None = None Used like this: lightly_train.train_image_classification( out="out/my_experiment", model="dinov3/vitt16", data={...}, model_args={"class_weights": "auto"}, # or: model_args={"class_weights": {"cat": 1.0, "dog": 3.5}}, ) "auto" needs a different formula per classification task, because the two tasks use different losses: multiclass -> CrossEntropyLoss(weight=...), using inverse class frequency scaled so the mean weight is 1.0. This balances the classes against each other. mu

The ratio argument for the random resized crop is not exposed, i.e. can't be wired through which should be fixed for the pretraining pipeline.

Problem EoMT models run out of GPU memory in the validation step. Training with the same batch size works. Issue #936 reports this. A user trains dinov3/vitt16plus-eomt-inst-coco with batch_size=1 on a 12 GB card. Training uses about 2.5 GB. Validation stops with torch.OutOfMemoryError: Tried to allocate 3.15 GiB. We want to lower the memory use. The metric values must stay the same. Cause The validation step resizes the predicted masks to the original image size. This follows the EoMT evaluation protocol. The step resizes all queries together. Here are the numbers from #936. The model has 200 queries. The original image is 2943x1435 pixels. One float32 tensor of shape (1, 200, 1435, 2943) needs 3.15 GiB. This is the exact number in the error message. get_labels_masks_scores then makes four more tensors of the same shape: masks, masks_fp32, the output of sigmoid(), and their product. The peak is near 13 GiB. Training does not have this problem. Training resizes the masks to the model i

Description Dense prediction tasks such as semantic segmentation and object detection need normalized intermediate feature maps from multiple backbone layers or stages. Multi-scale features can also be useful during pretraining and distillation, for example to define objectives or transfer representations at more than one level of the network, e.g. wanted in #628 . LightlyTrain already defines a public interface for this: ForwardMultiScaleFeatures, MultiScaleFeatureDims, and MultiScaleFeatureStrides MultiScaleFeatureViT and MultiScaleFeatureCNN MultiScaleFeaturePackage DINOv2 and DINOv3 provide reference implementations: DINOv2 ViT wrapper DINOv3 ViT wrapper DINOv3 ConvNeXt wrapper We would like to extend this interface to the remaining model packages, starting with TIMM. The current TIMMModelWrapper already uses TIMM's forward_intermediates where available, which should provide a useful starting point. EdgeCrafter is partially supported: ECViTModelWrapper.forward() already returns