apache/datafusion-ballista

Is your feature request related to a problem or challenge? Please describe what you are trying to do. We do not currently check for unused dependencies anywhere in CI. Our Rust lint surface is cargo fmt, cargo clippy, cargo tomlfmt, plus the docs/proto/config-docs checks, and the Dependencies workflow only covers the MSRV check and the python/Cargo.lock sync check. As a result, dependencies that are no longer used stay in our manifests. They still get resolved, downloaded, and (in many cases) compiled, which slows down builds, adds noise to Cargo.lock, and grows the surface we have to audit for supply chain and license reasons. Running cargo machete locally today reports quite a few: ballista-core -- ./ballista/core/Cargo.toml: aws-config aws-credential-types md-5 prost-types ballista-scheduler -- ./ballista/scheduler/Cargo.toml: insta prost-types tonic-prost ballista-executor -- ./ballista/executor/Cargo.toml: arrow tracing pyballista -- ./python/Cargo.toml: async-trait msrv

Both ballista-core and ballista-executor declare a feature named arrow-ipc-optimizations, and both have it on by default: ballista/core/Cargo.toml:38, defaulted at line 40 ballista/executor/Cargo.toml:36, defaulted at line 46 They are unrelated features that happen to share a name. The executor's gates exactly one call site, flight_service.rs:159. Core's gates the rest of the shuffle read path. The executor's does not forward to core's, unlike spark-compat right next to it at ballista/executor/Cargo.toml:47. This makes ballista/client/Cargo.toml:33-35 misleading. It pulls the executor with default-features = false, features = ["arrow-ipc-optimizations"], which reads as "keep the IPC optimization on" but only enables the executor's half. Core's happens to be on anyway through the client's direct dependency on ballista-core at line 32, so nothing is broken today, but it works by accident. Anyone building the executor standalone with --no-default-features --features arrow-ipc-optimizati

For REST, could we use: OpenAPI to document our format to clients? utoipa to generate OpenAPI spec from our Rust types we serde_json oasdiff in CI to prevent breaking changes This would enable strict enforcement and formally describe our API Originally posted by @avantgardnerio in #2261 Initially we can start from scheduler API

Is your feature request related to a problem or challenge? Please describe what you are trying to do. ExecutorMetadata contains ExecutorSpecification & ExecutorOperatingSystemSpecification pub struct ExecutorMetadata { /// Unique executor identifier. pub id: String, /// Hostname or IP address of the executor. pub host: String, /// Port number for data transfer. pub port: u16, /// Port number for gRPC communication. pub grpc_port: u16, /// Resource specification for this executor. pub specification: ExecutorSpecification, /// OS and hardware info for this executor pub os_info: ExecutorOperatingSystemSpecification, } where ExecutorOperatingSystemSpecification has more data (quite few of them strings pub struct ExecutorOperatingSystemSpecification { /// System name pub system_name: String, /// Kernel version pub kernel_ver: String, /// OS version pub os_ver: String, /// OS version (long) pub os_ver_long: Str

Is your feature request related to a problem or challenge? Please describe what you are trying to do. stage plan gets a too busy currently plan is generated at datafusion-ballista/ballista/scheduler/src/api/handlers.rs Line 373 in 3d9c1b5 let stage_plan = format!("{:?}", graph); I believe, cleaning it up may benefit EXPLAN ANALYZE as well Describe the solution you'd like change/cleanup the output format make it less cluttered and easier to follow and understand Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. Additional context Add any other context or screenshots about the feature request here.