#web-scraping (30 Repositories)
Ranked open-source repositories tagged with #web-scraping, scored by pull request acceptance likelihood and maintainer engagement velocity.
72.7%
39.6h
30 repositories tagged #web-scraping
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
konippi/servo-fetch
A self-contained browser engine that fetches, renders, and extracts web content as Markdown, JSON, or screenshots — no Chromium, no API key, no setup.
MODSetter/SurfSense
Air gapped, privacy focused open source NotebookLM alternative. Join our Discord: https://discord.gg/ejRNvftDp9
only-cli/oc
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
austin-weeks/miasma
Trap AI web scrapers in an endless poison pit.
microlinkhq/browserless
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
alexdlaird/amazon-orders
A Python library (and CLI) for Amazon order history, line items, and transactions
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
pinchtab/pinchtab
High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
Manavarya09/design-extract
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
1broseidon/ketch
Fast, stateless CLI for web search and scrape. Built for AI agents.
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
getmaxun/maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
jordantete/OddsHarvester
A python app designed to scrape and process sports betting data directly from oddsportal.com 🎯
axsddlr/vlrggapi
An Unofficial REST API for vlr.gg, a site for Valorant Pro Esports match results and news.
TickTockBent/charlotte
Token-efficient browser MCP server — structured web pages for AI agents, not raw accessibility dumps
jaebradley/basketball_reference_web_scraper
NBA Stats API via Basketball Reference
apify/actor-scraper
House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
CloakHQ/CloakBrowser
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
clearcotelabs/clearcote-browser
Open-source stealth Chromium with engine-level fingerprint spoofing - de-Googled, drop-in Playwright, fully buildable and verifiable from source.
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
tinyfish-io/agentql
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
feder-cr/invisible_playwright
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
zyocum/reader
Extract clean(er), readable text from web pages via trafilatura.
Anakin-Inc/anakin
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up