delta-io/delta-rs

What happened? We use deltalake with the datafusion and s3 features in the ModelarDB project hosted here. After updating to deltalake v1.0.0, our builds started failing on macOS with the error error[E0275]: overflow evaluating the requirement. I also tried building ModelarDB with deltalake v1.0.0 on Arch Linux, FreeBSD, and Microsoft Windows. In all cases, the build finished without any issues. On macOS it seems the issue can be fixed by adding a type annotation to Line 93 in delta-rs/crates/core/src/delta_datafusion/data_validation.rs so let validations = validations becomes let validations: Vec<Expr> = validations as shown here and in this patch. diff --git a/crates/core/src/delta_datafusion/data_validation.rs b/crates/core/src/delta_datafusion/data_validation.rs index 11a072c4..08120b73 100644 --- a/crates/core/src/delta_datafusion/data_validation.rs +++ b/crates/core/src/delta_datafusion/data_validation.rs @@ -90,7 +90,7 @@ impl DataValidation { input: LogicalPlan,

Is your feature request related to a problem? No response Describe the solution you'd like Hey now that we have a fancy 1.0, need our CI to do proper semantic versioning checks against it! Describe alternatives you've considered No response Priority None Additional context No response Contribution I'm willing to submit a pull request for this feature I can help with testing this feature I can help with documentation for this feature

Is your feature request related to a problem? #4763 this PR adds lazy collect on the table history,. Describe the solution you'd like In Python it is still materialized in full before returning. We should return it as a generator for python users Describe alternatives you've considered No response Priority None Additional context No response Contribution I'm willing to submit a pull request for this feature I can help with testing this feature I can help with documentation for this feature

Is your feature request related to a problem? What happened? DeltaTable.optimize.compact() changes the physical row order of Parquet files whose rows were already sorted by a timestamp column. The reproducer below writes monotonically increasing timestamp batches to a Delta table partitioned by calendar month. Before compaction: Every input batch is sorted by dt. The timestamp ranges of consecutive batches do not overlap. Every active Parquet file is internally sorted by dt. Reading the active files in partition and max.dt order produces a globally sorted sequence. After calling table.optimize.compact(), some compacted Parquet files are no longer internally sorted. Consequently, the final assertion fails. This is especially surprising because the returned compaction metrics report: preserveInsertionOrder: True preservedStableOrder: True plannerStrategy: preserveLocality The reproducer explicitly reads each Parquet file in physical row order through a DataFusion external table. It do

Is your feature request related to a problem? Feature Request Expose a Python transaction API that can commit AddAction and RemoveAction together in a single Delta transaction. Use Case I am building a Python service that ingests CSV logs into Delta Lake tables. For append-only writes, DeltaTable.create_write_transaction() works well because I can write Parquet files myself and commit the generated AddActions. However, for custom compaction / file rewrite workflows, I need to atomically commit both: AddActions for newly compacted Parquet files RemoveActions for old small files being replaced Currently the Python API appears to accept only AddActions: DeltaTable.create_write_transaction( actions: list[AddAction], ... ) This is enough for append-only writes, but not for compaction. If old files cannot be removed in the same transaction, queries will read both old and new files and produce duplicate data. Describe the solution you'd like Proposed Solution Expose a more general P