apache/datafusion-comet

Describe the bug With the native from_json path enabled (spark.comet.expression.JsonToStructs.allowIncompatible=true), a schema that repeats a field name fails the task once the result reaches the JVM. Spark accepts such a schema and fills the last field with that name, so from_json('{"a":1}', 'a INT, a INT') returns {null, 1}. Comet's native projection builds the same struct, and to_json over it matches Spark, but importing the result on the JVM fails because Java Arrow keys struct children by name: java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1) at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562) at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92) at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68) at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62) at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264) at org.apache.comet.vector.NativeUtil.getNextBatch(

Describe the bug A cast whose target struct type repeats a field name fails the task when Comet runs it natively. Spark allows the cast as long as each pair of fields casts, so CAST(s AS STRUCT<x: INT, x: INT>) over a STRUCT<p: INT, q: INT> column returns {1, 10}. Comet's native cast_struct_to_struct builds the result with the target fields, and the projection itself works: to_json over the cast matches Spark with {"x":1,"x":10}. Importing the result on the JVM then fails, because Java Arrow keys struct children by name and the two x children collapse into one: java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1) at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562) at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92) at org.apache.arrow.c.ArrayImporter.importChild(ArrayImporter.java:83) at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:101) at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java

Describe the bug make format does not produce code that passes the repo's own scalastyle check, and can itself introduce the violation. make format runs scalafix:scalafix and spotless:apply (scalafmt). scalafmt (scalafmt.conf: maxColumn = 98, rewrite.rules = [Imports]) wraps a long brace-less if/else so that a branch spans multiple lines, but it never inserts braces — scalafmt has no brace-adding rewrite (RedundantBraces only removes them), and none of the configured scalafix rules (ExplicitResultTypes, NoAutoTupling, RemoveUnused, DisableSyntax, LeakingImplicitClassVal, NoValInForComprehension, ProcedureSyntax, RedundantSyntax) add them. scalastyle's IfBraceChecker (dev/scalastyle-config.xml, singleLineAllowed=true, doubleLineAllowed=true) then fails the build with If block needs braces, because a brace-less branch that spans multiple lines is neither single- nor double-line. Since make format does not run scalastyle, it reports success while leaving code that the build/CI rejects,

Part of #5572. named_struct with duplicate field names falls the whole projection back to Spark: if (expr.names.length != expr.names.distinct.length) { Unsupported(Some(duplicateNamesReason)) } spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43 docs/source/user-guide/latest/expressions.md records it as "Duplicate field names fall back". Spark permits duplicate field names in a struct and CreateNamedStruct.doGenCode builds the row positionally, so the dispatcher reproduces it exactly. StructType is supported by CometBatchKernelCodegen.isSupportedDataType recursively, and duplicate names are not a problem on the Arrow side — the kernel's output field construction is positional too. Worth confirming with a test that the resulting Arrow schema round-trips, since duplicate field names in an Arrow struct are legal but unusual. Fix: mix CodegenDispatchFallback into CometCreateNamedStruct. CometCreateNamedStruct also declines inside convert at :66 when a child fails to seriali

Part of #5572. CometStringTranslate reports Incompatible unconditionally: override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible( Some(incompatReason)) spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120 The reason is real — DataFusion's translate iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no CodegenDispatchFallback mixin, the consequence is that translate falls the whole projection back to Spark unless the user sets spark.comet.expression.StringTranslate.allowIncompatible=true and accepts wrong answers. Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See CometInitCap (:146), CometStringReplace (:177), CometCaseConversionBase for upper / lower (:50), and the regex family at :368, :470, :508, :545, :598. Fix: mix CodegenDispatchFallback into Com