Lightning-AI/litData

🐛 Bug When optimize(..., use_checkpoint=True) writes checkpoints to remote storage, checkpoint destination paths can be truncated and moved outside the requested output prefix. LitData 0.2.71 writes stable checkpoint filenames such as checkpoint-0.json. However, remove_uuid_from_filename() still assumes every file under .checkpoints ends in -<32-character UUID>.json and unconditionally removes the final 38 characters. For example: s3://bucket/output/data/train/.checkpoints/checkpoint-0.json -> s3://bucket/output/dat.json With bucket versioning enabled, repeated checkpoint updates create many versions of the same malformed object. To Reproduce Install LitData 0.2.71, or use current main at commit a847d57. Run the code sample below. No S3 credentials or network access are required because it only exercises the destination-path calculation used by the uploader. Observe that both paths lose the .checkpoints directory, the checkpoint filename, and part of the output prefix. Cod

Problem In StreamingRawDataset, the adaptive concurrency budget for remote downloads was previously calculated using hardcoded static assumptions: _ASSUMED_AGGREGATE_BANDWIDTH_BPS = 100 * 1024 * 1024 # ~100 MB/s _ASSUMED_REQUEST_RATE = 6000.0 # target req/s _ASSUMED_REQUEST_LATENCY_S = 0.040 # 40 ms RTT _DEFAULT_MEDIAN_FILE_BYTES = 256 * 1024 These assumptions do not reflect the large variation in real-world cloud environments. Low-Bandwidth / Shared Nodes: Multi-tenant EKS/Kubernetes nodes may provide only ~10–20 MB/s of effective bandwidth. The static model can overestimate sustainable concurrency, leading to HTTP 429 throttling, socket saturation, and connection timeouts. High-Bandwidth / High-Performance Storage: Dedicated A100/H100 nodes with multi-Gbps networking or low-latency object storage can support significantly higher throughput. The conservative static cap can unnecessarily limit concurrency, resulting in GPU underutiliza

Description Describe the bug When calling optimize(..., mode="append") with num_nodes > 1, pre-existing dataset chunk metadata is included in every node's intermediate {node_rank}-index.json file. When the master node merges these intermediate indices, the existing chunks are concatenated repeatedly, duplicating them N times in the final index.json, where N is the number of nodes. Intended: Existing [A, B] ├──► Node 0 [C] → 0-index.json ──┐ └──► Node 1 [D] → 1-index.json ──┴──► Merge → [A, B, C, D] Current Bug: Existing [A, B] ├──► Node 0 [A, B, C] → 0-index.json ──┐ └──► Node 1 [A, B, D] → 1-index.json ──┴──► Merge ↓ [A, B, C, A, B, D] Root Cause data_processor.py: Every node passes existing_index to _merge_no_wait(). writer.py: _merge_no_wait() adds existing_index["chunks"] to ea

🐛 Bug When optimizing a dataset of numpy arrays I get the stated warning that the array is not writable and that this can cause issues down the line. The warning gets printed once we start iterating a StreamingDataLoader. Triggered by torch\utils\data_utils\collate.py:288 Note that this warning is not raised when casting the numpy arrays to torch tensors before optimizing, so maybe this is a non-issue for this library. To Reproduce I simplified the example given in this repo by having random_images not return PIL Image but the underlying numpy instead. import numpy as np import litdata as ld import torch def random_images(index): fake_images = np.random.randint(0, 256, (32, 32, 3), dtype=np.uint8) # ! not a PIL image, but numpy array fake_labels = np.random.randint(10) # fake_images = torch.from_numpy(fake_images) # no warning when working with torch data = {"index": index, "image": fake_images, "class": fake_labels} return data if __name__ == "__ma

🐛 Bug Parquet tests fail inconsistently in CI, mostly due to timeout, waiting for index file to be available. Possibly, multiple parquet tests leaking the environment. To Reproduce Steps to reproduce the behavior... Code sample # Ideally attach a minimal code sample to reproduce the decried issue. # Minimal means having the shortest code but still preserving the bug. Alternatively, you can share a fully reproducible Lightning Studio environment: A simple guide on how to create such a studio can be found here. Create a Studio. Reproduce the issue in the Studio. Publish the Studio. Paste the Studio link here. Expected behavior Additional context Environment detail PyTorch Version (e.g., 1.0): OS (e.g., Linux): How you installed PyTorch (conda, pip, source): Build command you used (if compiling from source): Python version: CUDA/cuDNN version: GPU models and configuration: Any other relevant information: