#web-crawler (27 Repositories)
Ranked open-source repositories tagged with #web-crawler, scored by pull request acceptance likelihood and maintainer engagement velocity.
38.1%
45.2h
27 repositories tagged #web-crawler
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
lexmount/moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
apache/stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
apache/nutch
Apache Nutch is an extensible and scalable web crawler
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
crawler-commons/crawler-commons
A set of reusable Java components that implement functionality common to any web crawler
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Goldziher/basemind
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
Anakin-Inc/anakin
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
MarginaliaSearch/MarginaliaSearch
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
commoncrawl/news-crawl
News crawling with StormCrawler - stores content as WARC
tech-engine/goscrapy
GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.
spider-rs/spider
Foundational low latency web data collecting in Rust
PhialsBasement/LibreCrawl
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
s0rg/crawley
The unix-way web crawler
lewisdonovan/google-news-scraper
Lightweight scraper for Google News
VIDA-NYU/ache
ACHE is a web crawler for domain-specific search.
crawlab-team/crawlab-lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
infinilabs/crawler
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
xianhu/PSpider
简单易用的Python爬虫框架，QQ交流群：597510560
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
internetarchive/Zeno
State-of-the-art web crawler 🔱
sjdirect/abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.