substrait-io/substrait

Follow-up to this review thread on #1148, filed with @benbellick's agreement to handle it separately rather than growing that fix. Two test modules independently discover and walk the checked-in extension YAML files. The duplication predates #1148 but that PR made it more visible. Updated 2026-08-05 after merging #953 into #1148 — see the comment below. find_extension_files no longer filters on the functions_ prefix, so the two discovery helpers are now closer than originally described and design question 2 has largely answered itself. 1. File discovery find_extension_files(dir_path) extension_yaml_files() location tests/coverage/extensions.py tests/type/test_type_grammar.py scope every .yaml under the given root every extensions/*.yaml, plus site/examples/**/*.yaml walk os.walk, recursive, caller-supplied root Path.glob, repo root derived from __file__ returns sorted list[str] sorted generator of Path consumers read_substrait_extensions(), validate_nullability_mark

Summary add and subtract in extensions/functions_datetime.yaml each carry a single function-level description that is applied to six or seven impls whose semantics differ materially. Several of those differences are not recoverable from the signature alone. Since impls[].description is already supported by the schema, these should be documented per impl. The schema already allows this description is a permitted property on each entry of impls for scalar, aggregate and window functions (text/simple_extensions_schema.yaml, in the impls.items.properties blocks). There is existing precedent in count (functions_aggregate_generic.yaml#L5-L26), which uses the function-level description as an umbrella and impl-level descriptions to disambiguate: - name: "count" description: Count the number of values or records. impls: - description: Count the non-null values in a set of values. ... - description: Count the number of records (not field-referenced). ... Wha

Split out of #771 (item 3 of that issue's "Desired State" list); the other items there are now resolved. Current behaviour Extension test coverage is enforced by assertions inside a pytest test, test_substrait_extension_coverage in tests/test_extensions.py: coverage.num_tests_with_no_matching_function == 0 Baseline.validate_against (tests/baseline.py) compares the computed numbers against the checked-in tests/baseline.json as a one-directional ratchet: the registry / test / covered-variant counts may only increase, and the number of function variants without coverage may only decrease. That test runs in CI via pixi run test (the python-style job in .github/workflows/pr.yml) and via the check-substrait-extensions_coverage pre-commit hook, so a coverage regression blocks both commits and merges. At the current baseline, 245 of 498 function variants have test coverage — 253 variants have none. What was asked for From #771: CI should block on test failures, but not on coverage changes

The simple extensions docs say that a dependency alias can be used: This alias can then be used as a .-separated namespace prefix wherever a type class or type variation name is expected. The "or type variation name" part of that sentence has no place where it can be applied. A type variation name is only ever written in two places: As a declaration, in type_variations[].name of the extension YAML that defines it. An alias prefix is meaningless there. As a reference from a plan, via extension URN + name in SimpleExtensionDeclaration, which does not use YAML dependency aliases at all. There is no production in grammar/SubstraitType.g4 that names a type variation, so an alias-prefixed variation name cannot appear in any type expression either. The only cross-extension reference a type variation can make is its parent, and that is a type class reference, which the sentence already covers. Suggested fix Narrow the sentence to type classes, e.g.: A YAML file can also reference type cla