opensearch-project/OpenSearch

Problem In the pluggable data format (composite engine — Parquet primary + Lucene secondary), PPL/SQL are first-class query interfaces. The per-shard execution concurrency is controlled by target_partitions — the number of parallel partitions DataFusion uses within each shard to scan Parquet segments, build hash tables, and run aggregations. More partitions = more parallelism = more memory; fewer = lower memory but slower single-query latency. Today, target_partitions is derived once at node level from cluster settings and baked into a WireConfigSnapshot that all queries share: // DatafusionSettings.java this.snapshot = WireConfigSnapshot.builder() .targetPartitions(deriveTargetPartitions(concurrentSearchMode, maxSliceCount)) ... .build(); The derivation uses search.concurrent.max_slice_count and search.concurrent_segment_search.mode: mode=none → target_partitions = 1 max_slice_count = 0 → availableProcessors / 2 otherwise → min(max_slice_count, availableProcesso

Is your feature request related to a problem? The experimental pluggable data format (sandbox) — the composite engine with Parquet primary + Lucene secondary — has no getting-started documentation. New contributors and developers who want to try the feature have to piece together instructions from code and comments. Proposed solution Add a Getting Started guide at sandbox/plugins/composite-engine/GETTING_STARTED.md and link it from the main DEVELOPER_GUIDE.md. The guide should cover: What the pluggable data format is — the composite engine (Parquet primary + Lucene secondary) Running OpenSearch from source with the data-format plugins via ./gradlew run -PinstalledPlugins='[...]' (auto-enables the experimental feature flag and native lib path) Creating a composite-backed index using index.pluggable.dataformat* / index.composite.* settings, ingesting data, and verifying (including the _count-returns-0 gotcha on a Parquet-primary index) Running sandbox test tiers: unit (test), inte

Problem When running OpenSearch locally via ./gradlew run with sandbox plugins that depend on the native Rust library (libopensearch_native.so/.dylib), the build system does not check whether the compiled native artifact matches the current Rust source. This means: If Rust FFI source has changed but buildRustLibrary has not been re-run, OpenSearch boots with a stale native library This causes silent wrong behavior, missing FFI exports (UnsatisfiedLinkError), or crashes that are difficult to diagnose Developers must manually remember to rebuild the Rust library — there is no staleness warning or automatic rebuild Expected behavior When any sandbox plugin declaring a native dependency is included in the installedPlugins list, gradlew run should either: Option A: Automatically trigger :sandbox:libs:dataformat-native:buildRustLibrary if Rust sources are newer than the compiled artifact (preferred) Option B: At minimum, emit a warning at startup if the .so/.dylib timestamp is older tha

Describe the bug Test testQtfFetchCancelTearsDownAndCleansUp of AnalyticsQueryTaskCleanupIT suite fails intermittently with below error AnalyticsQueryTaskCleanupIT > testQtfFetchCancelTearsDownAndCleansUp FAILED java.lang.AssertionError: injected failure on indices:data/read/analytics/fetch_by_row_ids must fail the QTF query, not return (response=org.opensearch.ppl.action.PPLResponse@6d6326e9) at __randomizedtesting.SeedInfo.seed([47CD2BA433B27B2A:CE953F2EA8F7C393]:0) at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertNotNull(Assert.java:713) at org.opensearch.analytics.cancellation.AnalyticsQueryTaskCleanupIT.assertQtfInjectedFailureSurfacesAndCleansUp(AnalyticsQueryTaskCleanupIT.java:611) at org.opensearch.analytics.cancellation.AnalyticsQueryTaskCleanupIT.testQtfFetchCancelTearsDownAndCleansUp(AnalyticsQueryTaskCleanupIT.java:580) REPRODUCE WITH: ./gradlew ':sandbox:qa:

Describe the bug A workload group with resiliency_mode: monitor still rejects search requests with HTTP 429 when its resource limit is breached. monitor is documented as observe-only ("it will not cause any cancellation but just log the eligible task cancellations"), so it should not reject live traffic. The cause is in WorkloadGroupService.rejectIfNeeded: the only resiliency-mode exemption is for SOFT (skipped unless the node is in duress). MONITOR falls through to the same rejection path as ENFORCED, throwing OpenSearchRejectedExecutionException and incrementing total_rejections. This makes monitor self-contradictory: it is correctly dormant on the cancellation path (only ENFORCED groups are passed to cancelTasks), but fully enforcing on the rejection path. Related component Search:Resiliency To Reproduce Enable WLM: wlm.workload_group.mode: enabled, and lower wlm.workload_group.node.cpu_rejection_threshold (e.g. 0.1) to make a breach easy to trigger. Create a workload group with re