#crawling (30 Repositories)
Ranked open-source repositories tagged with #crawling, scored by pull request acceptance likelihood and maintainer engagement velocity.
35.1%
46.0h
30 repositories tagged #crawling
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
dondai44423/donsetch
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block.
apache/nutch
Apache Nutch is an extensible and scalable web crawler
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
NateScarlet/holiday-cn
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
hardkoded/puppeteer-sharp
Headless Chrome .NET API
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
exa-labs/exa-mcp-server
Exa MCP for web search and web crawling!
ArchiveBox/abx-dl
⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...
xberg-io/crawlberg
High-performance web crawling engine with bindings for 11 languages
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
StJudeWasHere/seonaut
Open source SEO audit tool.
codelucas/newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
l4rm4nd/LinkedInDumper
Python 3 script to dump/scrape/extract company employees from LinkedIn API
infinilabs/crawler
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
mishakorzik/Infect
Create you virus in termux!
needleworm/bhban_rpa
<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.
crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
MarshalX/telegram-crawler
🕷 Automatically detects changes to the official Telegram sites, beta clients, MTProto servers and mini apps
elixir-crawly/crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
janreges/siteone-crawler
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
zorlan/skycaiji
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
lumpinif/deepcrawl
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.
clemfromspace/scrapy-selenium
Scrapy middleware to handle javascript pages using selenium
roach-php/core
The complete web scraping toolkit for PHP.