apache/hudi

#16639 wrote down where schema helpers belong and removed the duplication inside the Avro/HoodieSchema util classes themselves. Two look-alike pairs are still standing outside them, and both are the failure mode that issue describes: the same logic re-implemented because the existing one was not found. 1. generateProjectionSchema, implemented twice HoodieSchemaUtils#generateProjectionSchema(HoodieSchema, List<String>) -- hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:496 HoodieRealtimeRecordReaderUtils#generateProjectionSchema(HoodieSchema, Map<String, HoodieSchemaField>, List<String>) -- hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java:124 Same loop, same case-insensitive lookup, same exception down to the punctuation: "Field " + fn + " not found in log schema. Query cannot proceed! Derived Schema Fields: " The wording shows the copy direction: the general-purpose hudi-common helper still says "log schema

Describe the problem HoodieProcedureFilterUtils.evaluateFilter resolves function calls against a hardcoded resolution table; any function not in the table falls through as case _ => unresolvedFunc (HoodieProcedureFilterUtils.scala:353), stays unresolved, throws on eval inside the per-row Try, and the row is treated as a non-match. validateFilterExpression (:461) only validates column references, so no error ever surfaces: a typo'd or unsupported function yields an empty result set that is indistinguishable from "no matching rows". Probe (Spark 4.1.1): concat(name, 'x') = 'a1x' keeps 0/1 rows instr(name, 'a') = 1 keeps 0/1 if(name = 'a1', true, false) keeps 0/1 (while case when name = 'a1' then true else false end keeps 1/1, because the parser lowers it without an UnresolvedFunction) control: upper(name) = 'A1' keeps 1/1 Reachable from every procedure that exposes a filter argument. To reproduce Call any filter-supporting procedure with a function outside the table, e.g. show_cleans(t

Describe the problem HoodieProcedureFilterUtils.convertValueToInternal converts array columns (Seq / Array / java.util.List) to a raw Object[] instead of Catalyst ArrayData. Evaluating any predicate touching an array column -- size(...), array_contains(...), even isnotnull(col) -- throws ClassCastException inside BoundReference.eval, which the per-row Try swallows to false, so every row is dropped. The sort_array / array_size entries in the function-resolution table are dead code. (Maps are unaffected: they go through ArrayBasedMapData.) To reproduce Pinned in TestHoodieProcedureFilterUtils ("converts array / decimal / binary / uuid / java-time columns") since #19161: size(arrScala) >= 0 and isnotnull(arrScala) both return no rows. Suggested fix Wrap array values in new GenericArrayData(...) in convertValueToInternal, then flip the pinned assertions.

Describe the problem HoodieProcedureFilterUtils.applyTypeCoercion special-cases exactly one shape: a Long output column compared against an Int literal, with the column on the left of the comparison. Three problems fall out of this: Narrowing direction (original report). The Long-vs-Int case casts the column down to Int (Cast(boundRef, IntegerType)) instead of widening the literal to Long. For values beyond Int range this gives wrong results: under non-ANSI Spark the value wraps (false positives/negatives), under ANSI Spark the overflow error is swallowed by evaluateFilter's per-row Try and the row is silently dropped. Reversed operands never coerce. The match requires (BoundReference, Literal) in that order, so filter => "1500 < ts" evaluates Literal < BoundReference, falls through to case _ => original, fails to evaluate, and every row is dropped. Probe (Spark 4.1.1): ts > 1500 keeps 1/2 rows, 1500 < ts keeps 0/2, 1000 = ts keeps 0/2. Every other numeric type pair is uncoerced

hudi-presto-bundle was the first of three bundles that depend on hudi-hadoop-mr-bundle, a shaded fat jar, while shading the same library classes themselves. #19469 / #19490 handled that one. Two remain: packaging/hudi-hive-sync-bundle/pom.xml:167 packaging/hudi-gcp-bundle/pom.xml:157 Both list org.apache.hudi:hudi-hadoop-mr and org.apache.hudi:hudi-hadoop-common in their own artifactSet, so they shade those classes directly and have no need for the fat jar as a dependency. Why this is not simply the presto change again In the presto bundle, hudi-hadoop-mr was reached only through hudi-hadoop-mr-bundle's published dependency-reduced POM, which is what made #19433's regression possible: removing the fat jar emptied the includes and shade did not complain. These two bundles are in a better position — both reach the library modules through hudi-hive-sync, a library: $ mvn -o dependency:tree -Dincludes=org.apache.hudi -pl packaging/hudi-hive-sync-bundle,packaging/hudi-gcp-bundle hudi-h