NVIDIA-NeMo/Curator

Several heuristic filters in nemo_curator/stages/text/filters/heuristic/string.py crash when a document is empty or whitespace-only, instead of returning a score. from nemo_curator.stages.text.filters.heuristic.string import MeanWordLengthFilter MeanWordLengthFilter().score_document("") # ZeroDivisionError MeanWordLengthFilter().score_document(" ") # same for "\n\n" and "\xa0" Affected (nemo-curator 1.3.0; string.py on main is unchanged): Filter Error Line on main SymbolsToWordsFilter ZeroDivisionError num_symbol_words / len(words) (87) BulletsFilter ZeroDivisionError num_bullet_lines / len(sentences) (169) LongWordFilter ValueError: max() iterable argument is empty max(len(w) for w in ...) (235) MeanWordLengthFilter ZeroDivisionError sum(word_lens) / len(word_lens) (328) PunctuationFilter ZeroDivisionError ... / len(sentences) (351) EllipsisFilter ZeroDivisionError ... / len(sentences) (378) WordsWithoutAlphabetsFilter ZeroDivisionError num_english_alpha / len(

Problem When using RayDataExecutor with GPU stages, Ray Data's autoscaler starts with min_concurrency=1 and scales up one actor at a time as it measures throughput. In a benchmark with 4 GPUs and a single GPU stage, this means: Time GPUs active t=40s 1/4 t=81s 2/4 t=121s 3/4 t=161s 4/4 161 seconds are wasted ramping up before the pipeline runs at full utilization. By contrast, Xenna (which pre-allocates all actors upfront) reaches 4/4 GPUs by ~60s. Root cause calculate_concurrency_for_actors_for_stage in utils.py returns (1, max_actors) — a min of 1 — which tells Ray Data to start with one actor and autoscale up from there. Proposed fix Look into if some config from Ray Data's Config influences this to be more aggressive Since ray-project/ray#54648 this should be supported Impact On a 4-GPU, 1-GPU-stage benchmark (embeddinggemma-300m on 1GB tinystories), wall-clock time was 538s with Ray Data vs 428s with Xenna — roughly a 20% gap largely attributable to the slo

Problem RayDataStageActorAdapter calls setup_on_node and setup in __init__ but never calls teardown(). Stages that hold GPU resources (e.g. vLLM EngineCore subprocesses) are never cleaned up, causing GPU memory leaks between pipeline runs. Fix Per Ray's actor cleanup docs we might have to have __ray_shutdown__ def __ray_shutdown__(self): with suppress(Exception): self.teardown() RayActorPoolExecutor already does this correctly by calling actor.teardown.remote() before ray.shutdown(). Please make sure to add a unittest for this for instance you can add a test that creates a file on the filesystem and then deletes it in teardown.

Internally we can use swiftstack. But the benchmark should test CPU i/o for json / parquet GPU i/o for json / parquet (compressed / uncompressed) The findings should be reported and new issues be created

On Ray Data pipelines we specify concurrency and it seems like its deprecated since 2.51 2026-02-18 10:06:27,932 WARNING util.py:624 -- The argument ``concurrency`` is deprecated in Ray 2.51. Please specify argument ``compute`` instead. For more information, see https://docs.ray.io/en/master/data/transforming-data.html#stateful-transforms.