Eventual-Inc/Daft

Is your feature request related to a problem? We currently only support the SUBSTRING(str, start, len) syntax, not the ANSI-based SUBSTRING(str FROM start FOR len) syntax. Describe the solution you'd like We should just support both syntaxes. Our parser, sqlparser does seem to support both (https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Expr.html#variant.Substring), we just don't allow the ANSI case. See src/daft-sql/src/planner.rs::SQLPlanner::plan_expr for more details. Describe alternatives you've considered No response Component(s) No response Additional Context No response Would you like to implement a fix? No

Is your feature request related to a problem? We currently don't support the CROSS JOIN SQL syntax despite having an execution operator for cross joins. Describe the solution you'd like We should add support. Should be mainly a change in src/daft-sql/src/planner.rs::SQLPlanner::plan_single_from where we plan the other joins. Describe alternatives you've considered No response Component(s) Other Additional Context No response Would you like to implement a fix? No

Is your feature request related to a problem? Right now the SQL interfaces use other libraries to read from databases: daft.read_sql uses ConnectorX when available and SQLAlchemy otherwise. ConnectorX is compatible with a subset of databases, and even then compatibility is complicated. For example, see #7199. SQLAlchemy is more generic and pluggable but slow daft.write_sql only uses SQLAlchemy (ConnectorX only supports reads). Can be slow and not always that compatible with some of our complex types Describe the solution you'd like ADBC is a common interface for reading and writing Arrow-formatted data to databases. It should be similar in performance to ConnectorX with the pluggable extensibility of SQLAlchemy. We should support that in our APIs and then benchmark against ConnectorX and SQLAlchemy. If ConnectorX doesn't provide that much benefit, then lets drop it. Lets keep this in Python for now, I don't think it needs to be in Rust. Describe alternatives you've considered No resp

Is your feature request related to a problem? Right now, when implementing custom DataSources, particularly DataSourceTasks, you're forced to either: Write custom DataSourceTask Python code Use an existing built-in file format (Parquet, CSV, JSON, WARC, etc), which mainly applies to table formats like Iceberg This is kind of annoying, because DataSource is a useful abstraction for formats with metadata, like LeRobot, WebDatasets, DROID, etc. For most of these though, we usually just write DataFrame code to actually read the data in, which is way more flexible. Describe the solution you'd like Have a variant pf DataSources that return or wrap DataFrame creation code. Most of the DataSource semantics remain the same, the only difference is that the tasks are not DataSourceTasks, they end up being a single dataframe. That DataFrame replaces the node in the plan. For example, say that we have a LeRobotDataSource. With it, we can: Perform count, limit, and aggregation pushdowns on the JS

Is your feature request related to a problem? Similar to #7342, there's no way to write WebDatasets, which is pretty common to use for training. Describe the solution you'd like Support writing WebDatasets. Similar to the read support, I think this only really needs to be a DataSink. Describe alternatives you've considered No response Component(s) Data Sources & Sinks Additional Context No response Would you like to implement a fix? No