#spark (30 Repositories)
Ranked open-source repositories tagged with #spark, scored by pull request acceptance likelihood and maintainer engagement velocity.
70.2%
13.3h
30 repositories tagged #spark
ascii-supply-networks/dagster-slurm
Dagster SLURM integration
apache/zeppelin
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
apache/datafusion-comet
Apache DataFusion Comet Spark Accelerator
kamu-data/kamu-cli
Next-generation decentralized data lakehouse and a multi-party stream processing network
NVIDIA/cudf-spark
NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs
neo4j/neo4j-spark-connector
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
apache/paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
apache/auron
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
delta-io/delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
JohnSnowLabs/spark-nlp
State of the Art Natural Language Processing
microsoft/SynapseML
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
aehrc/pathling
Tools that make it easier to use FHIR and clinical terminology within data analytics, built on Apache Spark.
starlake-ai/starflow
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
awslabs/data-on-eks
DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS
projectnessie/nessie
Nessie: Transactional Catalog for Data Lakes with Git-like semantics
FirelyTeam/spark
Firely and Incendi's open source FHIR server
lakesoul-io/LakeSoul
LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.
databrickslabs/dqx
Databricks framework to validate Data Quality of pySpark DataFrames and Tables
apache/incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
smart-data-lake/smart-data-lake
Smart Automation Tool for building modern Data Lakes and Data Pipelines
kubeflow/spark-operator
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
broadinstitute/gatk
Official code repository for GATK versions 4 and up
graphframes/graphframes
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
apache/livy
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
RoaringBitmap/RoaringBitmap
A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others
exacaster/lighter
REST API for Apache Spark on K8S or YARN
yeasy/docker_practice
最新Docker容器技术，从真实案例中学习最佳实践！| Learn and understand Docker&Container technologies, with real DevOps practice!
zinggAI/zingg
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
RumbleDB/rumble
Quick start: pip install jsoniq ⛈️ RumbleDB 3.0.0 "Coast Redwood" 🌳 for Apache Spark | Run queries on your large-scale, messy datasets (JSON, text, CSV, Parquet, Delta...) | Data Lakehouse with Updates, Scripting, Declarative Machine Learning and more