lance-format/lance

Problem A Lance Namespace REST server return a standard throttling response: HTTP/1.1 429 Too Many Requests Retry-After: 2 with the official Namespace error code 21 (THROTTLING). However, the current REST Namespace client discards response headers while converting an HTTP response into a Namespace error. Callers only receive ThrottlingError with its message and error code; they cannot access Retry-After, the HTTP status, or other response metadata. This prevents SDK and application layers from implementing server-directed retry, queuing, or throttling coordination. Current behavior The REST request path in rust/lance-namespace-impls/src/rest.rs receives a reqwest::Response, reads the error body, and maps its Namespace error code to an error type. Response headers are not preserved during this conversion. Consequently, Python callers can catch: except lance_namespace.errors.ThrottlingError as error: ... but cannot determine whether: the server provided a retry delay; the throttling

We already have code that it generic over Utf8 and LargeUtf8, so it should mostly be a matter to findings the places where we match on those and add cases for StringView. Examples: lance/rust/lance-arrow/src/lib.rs Lines 470 to 476 in 11eeb29 pub fn iter_str_array(arr: &dyn Array) -> Box<dyn Iterator<Item = Option<&str>> + Send + '_> { match arr.data_type() { DataType::Utf8 => Box::new(arr.as_string::<i32>().iter()), DataType::LargeUtf8 => Box::new(arr.as_string::<i64>().iter()), _ => panic!("Expecting Utf8 or LargeUtf8, found {:?}", arr.data_type()), } } lance/rust/lance-index/src

In many places we output the same set of fields and/or schema per method call, or even per batch. This can be made cheaper if we initialize them once via a LazyLock and wrapping in an Arc. Here's an example where we construct a schema per query: lance/rust/lance-index/src/scalar/inverted/index.rs Lines 8107 to 8116 in 11eeb29 Field::new(FLAT_ALL_TOKENS_COL, DataType::UInt64, false), Field::new( FLAT_QUERY_TOKEN_COUNTS_COL, DataType::FixedSizeList( Arc::new(Field::new("item", DataType::UInt64, true)), query_tokens.len() as i32, ), false,

Follow up to #7226. Need to add that option for Java.

As we are wrapping up on the dataset level column statistics MVP, there are some high priority todo items: [ ] Support column stats generation for partial dataset. Currently the dataset will only be generated when all the files in the dataset has column stats written. A naive implementation is just to build the stats from the old fragment file rather than terminate early. [ ] Expose a Python API from the rust implementation. [ ] Exposing more stats in the discussion thread #4540 More could be found in the design doc