#scraping (30 Repositories)
Ranked open-source repositories tagged with #scraping, scored by pull request acceptance likelihood and maintainer engagement velocity.
78.6%
40.0h
30 repositories tagged #scraping
plabayo/rama
modular service framework to move and transform network packets
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
iiab/iiab
Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !
germondai/trawl
Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
apify/apify-sdk-python
Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.
firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
soxoj/socid-extractor
⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
daijro/camoufox
🦊 Anti-detect browser
feder-cr/Jobs_Applier_AI_Agent_AIHawk
Open source AI job application bot in Python: browser automation and web scraping to read job postings, then auto-apply with a tailored resume and cover letter for each posting.
sqdshguy/wreq-js
HTTP client for Node.js with browser TLS fingerprint impersonation
apify/apify-client-python
Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.
1broseidon/ketch
Fast, stateless CLI for web search and scrape. Built for AI agents.
playwright-php/playwright
Playwright PHP library for browser automation: navigation, E2E tests, assertions, screenshots, and so much more!
ralyodio/humanparser
Parse a human name string into salutation, first name, middle name, last name, suffix.
oxylabs/oxylabs-mcp
Official Oxylabs MCP integration
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
shurco/goClone
🌱 goClone - clone websites in seconds
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
umihico/docker-selenium-lambda
The simplest demo of chrome automation by python and selenium in AWS Lambda
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
axsddlr/vlrggapi
An Unofficial REST API for vlr.gg, a site for Valorant Pro Esports match results and news.
sparklemotion/mechanize
Mechanize is a ruby library that makes automated web interaction easy.
scrapy/parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
clearcotelabs/clearcote-browser
Open-source stealth Chromium with engine-level fingerprint spoofing - de-Googled, drop-in Playwright, fully buildable and verifiable from source.
tinyfish-io/agentql
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
feder-cr/invisible_playwright
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
masterFuf/taktik-bot
Instagram & TikTok automation via real Android devices. Likes, follows, DMs, scraping. No API abuse. Built with Python, uiautomator2 & ADB.
Lookyloo/lookyloo
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
spekulatius/PHPScraper
A universal web-util for PHP.