#data-pipeline (29 Repositories)
Ranked open-source repositories tagged with #data-pipeline, scored by pull request acceptance likelihood and maintainer engagement velocity.
64.2%
36.8h
29 repositories tagged #data-pipeline
datazip-inc/olake
OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.
rocketride-org/rocketride-server
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
Hebbian-Robotics/hflow
SDK for robotics teams to verify the quality of their data used for AI model training.
bruin-data/ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
ConduitIO/conduit
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
estuary/flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
dagucloud/dagu
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
DataSQRL/sqrl
Harness for building data engineering agents designed around human control, correctness, and safety.
superlinked/sie
Open-source inference server and production cluster for all the models your agent needs.
backblaze-labs/genblaze
Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for every output.
galaxy-io/filament
Pluggable data replication with checkpointing, batching, and integrity events
rocky-data/rocky
A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.
starlake-ai/starflow
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
debezium/debezium
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
Multiwoven/multiwoven
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
elementary-data/elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
ModelEngine-Group/DataMate
DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.
apache/flink-cdc
Flink CDC is a streaming data integration tool
clidey/whodb
Where data access meets operational intelligence
starlake-ai/starlake
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
slothflowlabs/duckle
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
lightfeed/extractor
Use LLMs to robustly extract web data
snowplow/snowplow
The leader in Customer Data Infrastructure
Feashliaa/job-board-aggregator
Job board aggregator indexing 1,000,000+ active positions from 20,000+ companies across Greenhouse, Lever, Ashby, Workday, and other major ATS platforms. Multithreaded Python ETL pipeline, daily automated refreshes via GitHub Actions, filterable client-side search.
dataflint/spark
Drop-in replacement for Apache Spark UI
ubisoft/mobydq
:whale: Tool to automate data quality checks on data pipelines
InfuseAI/piperider
Code review for data in dbt
superstreamlabs/memphis
Memphis.dev is a highly scalable and effortless data streaming platform
apache/seatunnel-web
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).