#data-engineering (30 Repositories)
Ranked open-source repositories tagged with #data-engineering, scored by pull request acceptance likelihood and maintainer engagement velocity.
75.9%
22.1h
30 repositories tagged #data-engineering
apache/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
datacontract/datacontract-cli
Open-source data contract testing for Snowflake, Databricks, BigQuery, Kafka & 15 more. Built on the Open Data Contract Standard (ODCS). MIT.
risingwavelabs/risingwave
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
xonsh/xonsh
🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.
mlrun/mlrun
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
ConduitIO/conduit
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
Zipstack/unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
dathere/qsv
Blazing-fast Data-Wrangling toolkit
benseverndev-oss/goldenmatch
Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.
estuary/flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
quiltdata/quilt
Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.
apache/superset
Apache Superset is a Data Visualization and Data Exploration Platform
DataSQRL/sqrl
Harness for building data engineering agents designed around human control, correctness, and safety.
treeverse/lakeFS
lakeFS - Data version control for your data lake | Git for data
DataRecce/recce
The data-validation toolkit for enhanced dbt (data build tool) PR review
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
hyparam/icebird
Icebird: JavaScript Iceberg Client
apache/devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
kanton-bern/hellodata-be
The Open-Source Enterprise Data Platform in a single Portal
airbytehq/airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
rocky-data/rocky
A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.
pyjanitor-devs/pyjanitor
Clean APIs for data cleaning. Python implementation of R package Janitor
starlake-ai/starflow
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
Eventual-Inc/Daft
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
bacalhau-project/bacalhau
Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.
dlt-hub/dlt
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
snowflakedb/snowpark-python
Snowflake Snowpark Python API
Multiwoven/multiwoven
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
sodadata/soda-core
Data Contracts engine for the modern data stack. https://www.soda.io