#pyspark (19 Repositories)
Ranked open-source repositories tagged with #pyspark, scored by pull request acceptance likelihood and maintainer engagement velocity.
24.3%
12.5h
19 repositories tagged #pyspark
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
narwhals-dev/narwhals
Lightweight and extensible compatibility layer between dataframe libraries!
JohnSnowLabs/spark-nlp
State of the Art Natural Language Processing
microsoft/SynapseML
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
apache/incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
G-Research/spark-extension
A library that provides useful extensions to Apache Spark and PySpark.
allisonwang-db/pyspark-data-sources
Custom PySpark Connectors
databrickslabs/dbldatagen
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
apache/linkis
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
sb-ai-lab/RePlay
A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models
mahmoudparsian/big-data-modeling
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
cartershanklin/pyspark-cheatsheet
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
titicaca/spark-iforest
Isolation Forest on Spark
microsoft/Azure-Databricks-NYC-Taxi-Workshop
An Azure Databricks workshop leveraging the New York Taxi and Limousine Commission Trip Records dataset
uber/petastorm
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
h2oai/sparkling-water
Sparkling Water provides H2O functionality inside Spark cluster
archivesunleashed/aut
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
awesome-spark/awesome-spark
A curated list of awesome Apache Spark packages and resources.
WeBankFinTech/Scriptis
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.