apache/arrow-rs

Is your feature request related to a problem or challenge? Follow-up to #5125. This issue captures the remaining test-organization work identified after the original module extractions. The test module in arrow-cast/src/cast/mod.rs contains approximately 11,400 lines and more than 300 tests. Most cast implementations have been divided into functional submodules, but their public-behavior tests remain collected in one large inline module. This accounts for most of the remaining size of cast/mod.rs and makes tests for individual cast families difficult to locate and maintain. Describe the solution you'd like Move the public-behavior tests currently in cast/mod.rs into a crate-level test suite, following the organization used by Parquet: arrow-cast/ ├── src/ ├── tests/ │ └── cast/ │ ├── mod.rs │ ├── binary.rs │ ├── decimal.rs │ ├── dictionary.rs │ ├── dispatch.rs │ ├── list.rs │

Is your feature request related to a problem or challenge? Follow-up to #5125. This issue captures the remaining implementation work identified after the original module extractions. Previous work extracted list, decimal, dictionary, string, map, run-array, and union functionality into private submodules. However, arrow-cast/src/cast/mod.rs still contains several cohesive groups of implementation helpers around the top-level cast kernel. This makes the module harder to navigate and leaves additional opportunities to continue the functional separation started by #5125. Describe the solution you'd like Move the remaining cohesive implementation helpers into private submodules while retaining cast_with_options as the centralized top-level dispatcher. Potential groups include: Temporal helpers for timestamps, dates, durations, intervals, and timezones Numeric and boolean conversion helpers Binary, fixed-size binary, and byte-view conversion helpers Struct conversion helpers Remaining

Is your feature request related to a problem or challenge? part of #9372 While working on the ALP encoder/decoder with @sdf-jkl in #9372, I found the parquet tests were spread out all over the place and thus is was hard to know where to put end to end tests and what was current covered or not covered. There are several categories of tests that appear frequently in the parquet crate and are currently scattered about "round trip" tests -- they write data to a parquet file and then read it back, ensuring the exact same values are returned "known fixture tests" -- read files from parquet-testing and assert that their contents match the expected values This approach directly tests the decoders, and transitively tests the encoders (since we know the decoder can read third-party files, we have some confidence that the writer is writing valid data as well) Describe the solution you'd like I would like to consolidate the end to end testing into arrow_reader and arrow_writer tests consolida

Is your feature request related to a problem or challenge? In some of our kernels we need to compute the scale for a decimal scale, via a call to pow(). We can replace such usages with calls to a precomputed lookup table of the powers per scale, to avoid a call to pow() at only the cost of some extra code. Describe the solution you'd like See example usage here: arrow-rs/arrow-cast/src/cast/mod.rs Lines 86 to 92 in ed92960 pub fn single_decimal_to_float_lossy<D, F>(f: &F, x: D::Native, scale: i32) -> f64 where D: DecimalType, F: Fn(D::Native) -> f64, { f(x) / 10_f64.powi(scale) } We could replace here with a lookup

Describe the bug The implementation for REE decoder forces the child field's decoder to accept nulls even though the Arrow schema for the values field may declare it non-nullable. As the REE encoding has no container-level null mask, this may smuggle null values into the values array that must not have them. Furthermore, even if the REE field itself is non-nullable, the flag passed down to construct the child decoder is ignored by e.g. StringArrayDecoder. The resulting nulls are not invalidated by a post-hoc check the way StructArrayDecoder does. To Reproduce Repro test case: #[test] fn test_read_run_end_encoded_nulls_in_non_nullable_values() { // A `RunArray` has no validity buffer of its own, so unlike a struct child // there is no parent-level mask that could make these nulls representable. // Declaring the enclosing field nullable does not change that. let ree_type = DataType::RunEndEncoded( Arc::new(Field::new("run_ends", DataTyp