Open source web scraping tools, installed and run

Clone any website with one command using AI coding agents

What the test found: The next.js app installs, builds, typechecks, lints, and runs with a dev server responding HTTP 200 on localhost:3000. 6 minutes.

Elegant Scraper and Crawler Framework for Golang

What the test found: installed Go 1.24.9, built colly v2.1.0, all tests passed, CLI binary functional. 1 minute.

Python ProxyPool for web spider

What the test found: The ProxyPool Flask API server runs on port 5010 backed by real Redis, responds to all endpoints (/get, /pop, /all, /delete, /count), and all 248 automated tests pass. 6 minutes.

A next-generation crawling and spidering framework.

What the test found: Katana v1.7.0 builds and runs: go test ./... passes all unit tests, the 5 functional tests (standard crawl, depth 3, headless, two authenticated recorded-flow variants) all pass, and the binary crawls a live HTTP target discovering URLs. 8 minutes.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

What the test found: newspaper3k 0.3.0 installs in a virtualenv on Python 3.12, all 40 unit tests pass, and core Article download/parse/nlp pipeline produces correct output from static HTML. 4 minutes.

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

What the test found: skill-seekers 3.10.0.dev0 installed and all 3857 unit/adaptor/scraper tests pass without external API keys or network dependencies. 6 minutes.

📊 Browser automation framework for scraping, testing, and completing tasks with Python. Supports pytest. Stealth options. Over 100 examples.

What the test found: SeleniumBase 4.54.7 is installed in a Python 3.12 venv; 8 framework unit tests, 7 offline HTML tests, and 2 real web tests passed using headless Chrome 154. 5 minutes.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

What the test found: Crawlab Go backend compiles, runs as master with MongoDB, serves the /system-info API endpoint on port 8080 returning 200, and its test suite runs clean for 4 core packages; the Vue frontend builds with Vite. 17 minutes.

A scalable web crawler framework for Java.

What the test found: All 8 Maven modules (webmagic-core, extension, scripts, selenium, saxon, samples, coverage) compile and their tests pass with 0 failures. The library is fully functional as a Java 11 crawler framework. 41 minutes.

Stealth headless browser for AI agents, bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

What the test found: Server starts with Camoufox browser under Xvfb, responds on port 9377, creates browser tabs that navigate to real URLs, returns accessibility snapshots with element refs, and processes click interactions correctly. All API endpoints function end-to-end. 4 minutes.

Crawlee, A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG...

What the test found: Dependencies installed with uv sync --all-extras, Playwright browsers (chromium + firefox) downloaded, all 2480 unit tests pass, lint and type-check pass, and the package builds and imports successfully at version 1.10.3. 8 minutes.

Lighter web automation with Python

What the test found: Helium 7.0.3 installed from source via pip install -e . Selenium 4.49.0 with Chrome 154 headless via Selenium Manager; all 332 tests pass with 2 regressions fixed (date input JS fallback, subprocess module discovery). 33 minutes.

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

What the test found: Package jmcomic 2.7.7 installed in a Python 3.12.3 venv with all dependencies; 238/240 tests pass including real network calls against JMComic API servers; CLI commands jmcomic and jmv produce correct help output. 10 minutes.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

What the test found: Trafilatura 2.2.0 is installed in editable mode in a .venv, all 428 tests pass, and both CLI and Python extraction work correctly on sample HTML input. 6 minutes.

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

What the test found: After installing the pre-built curl-cffi wheel (v0.16.3) into a fresh venv, the library imports, the CLI runs (curl-cffi get, doctor, --help), real HTTP requests with browser impersonation return 200, and the unit test suite passes 600+ tests across all 20 test modules including sync/async HTTP, websockets... 38 minutes.

Declarative data automation language and Go runtime for structured extraction workflows.

What the test found: Go 1.27.1 installed, all dependencies resolved, all 68 test packages pass with race detection, CLI binary compiles and runs FQL queries end-to-end. 7 minutes.

Redis-based components for Scrapy.

What the test found: scrapy-redis 0.9.1 is installed in a venv at /work/repo/.venv with real Redis 6.2 running on localhost:6379; all 102 tests pass covering connection pooling, dupefilter, queue (FIFO/LIFO/priority), scheduler, spider setup, idle backoff, multi-key consumption, and utility modules. 9 minutes.

🔮 A Node.js scraper for humans.

What the test found: scrape-it installs, builds, and passes its full test suite on Node 18.19.1 with cheerio pinned to 1.0.0. 3 minutes.

Every web site provides APIs.

What the test found: toapi 2.2.4 installed from source, all 7 tests pass, CLI help works, Flask server answers 200 with JSON, version 2.2.4 confirmed. 2 minutes.

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

What the test found: php-curl-class repository is fully installed with PHP 8.5.8 static binary and all Composer dependencies; the full test suite passes with 264 tests and 23390 assertions and zero failures. 43 minutes.

Node.js scraper to get data from Google Play

What the test found: The google-play-scraper npm package installs, all 84 tests pass against live Google Play, and the library's app/list/search/suggest/developer/reviews/similar/permissions/datasafety/categories methods all fetch and parse real data correctly. 8 minutes.

https://spatie.be/docs/crawler

What the test found: PHP 8.4.23 and Composer 2.7.1 installed from static binaries, all dependencies resolved via `composer update --prefer-stable`, and the full test suite passes (229/229) including both fake-HTTP unit tests and real-HTTP integration tests against a local PHP server. 6 minutes.

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

What the test found: The Geziyor web scraping framework builds and passes every test across all 7 packages, and a compiled demo successfully scraped a live website. 16 minutes.

Foundational low latency web data collecting in Rust

What the test found: Rust 1.99.0 installed, cargo build succeeds, spider CLI binary crawls and scrapes example.com returning status 200 and expected page content, all 608 spider core tests pass, workspace-wide crate tests all pass. 56 minutes.

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

What the test found: Build succeeded (mvn compile, package, test-compile all exit 0); demo crawler successfully fetched and parsed https://www.baidu.com/, producing JSON output. 5 minutes.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

What the test found: The CrawlerDetect PHP library installs, builds, and passes its full test suite of 28 tests with 170,886 assertions on PHP 8.0.30. 5 minutes.

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

What the test found: goclone binary builds from source, all 9 test suites across 3 packages pass, the built binary's help and version commands print correct output. 11 minutes.

Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.

What the test found: Stealth Browser MCP server runs on stdio transport, spawns a headless Chromium 157.0.8086.0, navigates to data URLs, extracts page HTML content, and closes cleanly -- all via MCP protocol over stdio. 15 minutes.

Web crawling framework based on asyncio.

What the test found: All 7 tests pass, ruff lint passes, library imports correctly, and item attribute extraction from HTML produces expected results. 1 minute.

Flexible Node.js crawler library

What the test found: x-crawl library builds successfully; its crawlData, crawlHTML, async/sync modes, baseUrl, timeout, errorCollect, proxy rotation, and Puppeteer browser APIs work correctly against a real HTTP server and CONNECT proxy. 34 minutes.

Async Python 3.6+ web scraping micro-framework based on asyncio

What the test found: Ruia 0.8.5 is installed in a Python 3.12 venv with uvloop; pip install -e .[uvloop] succeeds; 66 of 67 tests pass including all field extraction, item parsing, middleware, request fetching, and spider orchestration tests against live httpbin.org and news.ycombinator.com. 22 minutes.

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

What the test found: selectolax C extension successfully built, installed in a venv, all 279 tests passing, and basic HTML parsing + CSS selector usage verified. 4 minutes.

Undetected Python version of the Playwright testing and automation library.

What the test found: Patchright 1.63.0 is installed, Chromium browser is downloaded and launchable, the CLI reports its version, and 2237 of 2239 Chromium tests pass (sync + async). 37 minutes.

An ergonomic, privacy-aware Python HTTP Client

What the test found: wreq Python package builds, installs, and passes all 57 unit tests against a local httpbin server, supporting HTTP/1.1 and HTTP/2 requests, cookie jars, redirect policies, TLS emulation (JA3/JA4), multipart uploads, streaming, and compression. 15 minutes.

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

What the test found: Wombat 3.3.0 builds and installs as a gem under Ruby 3.2.3, all 56 RSpec tests pass, and a live Wombat.crawl scrape of a local WEBrick page returns correctly parsed structured data. 12 minutes.

🚀 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader, async REST API, MCP server...

What the test found: All 2960 tests pass (2450 unit + 510 integration), the CLI outputs version 5.0.1 and lists 10 commands, the API server starts and serves healthz/readyz/swagger/console root with 200, the web console SPA is built, and both the PostgreSQL+TimescaleDB and Redis backends and the Go downloader sidecar run. 48 minutes.

A Chrome DevTools Protocol driver for web automation and scraping.

What the test found: Rod compiles with go build ./... passes all vet checks, and launches Chromium 128 to control a browser page via DevTools Protocol. The main rod test suite and lib/cdp, lib/launcher (except known flaky TestManaged), and lib/proto sub-packages all pass. 59 minutes.

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

What the test found: The browserless monorepo installs under Node.js 24 with puppeteer 25, all packages build, Chrome 154 launches headlessly, and 390+ tests pass across 10 packages (browserless, goto, screenshot, capture, function, devices, errors, screencast, lighthouse, ai). 32 minutes.

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

What the test found: Crawl4AI 0.9.4 installed in venv, Playwright chromium browser downloaded, DB initialized, AsyncWebCrawler can crawl real URLs with markdown output, cache operations work, and 360 core tests pass with 11 skipped. 14 minutes.

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

What the test found: Firecrawl MCP Server v3.24.1 builds, installs, and starts successfully with 25 tools in stdio mode; 81 of 84 tests pass. 24 minutes.

👾 Fast and simple video download library and CLI tool written in Go

What the test found: Lux builds and runs: the binary outputs version info and help, extracts real YouTube video metadata, and 14 of 48 test packages pass (all failures are network-dependent extractor tests or DNS-timeout content-type lookups; no code defects remain). 11 minutes.

news-please - an integrated web crawler and information extractor for news that just works

What the test found: news-please 1.6.13 installs and its library API extracts structured article data (title, authors, publication date, description, maintext, language) from HTML via both HTTP and raw HTML input; CLI entry points respond with help output. 5 minutes.

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution...

What the test found: browser-act-cli 1.4.2 installed, skill handshake completed, and the chrome browser type (bundled Chromium) works end-to-end, browser create, open, navigate, state inspection, content extraction via get markdown, and session close all succeed; stealth features require a server-side-validated BrowserAct API key that... 5 minutes.

Scrapy, a fast high-level web crawling & scraping framework for Python.

What the test found: Scrapy 2.19.0 is installed and fully functional. The CLI, 4539/4540 tests, and an end-to-end crawl all work. The sole failing test is a container IPv6 limitation, not a code defect. 11 minutes.

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock...

What the test found: changedetection.io 0.60.7 installs into a venv, launches and serves HTTP 200, its full basic test suite passes (680 passed in the main group, exit 0, plus 508 unit tests), and a real watch added via the API detected a content change and exposed it in the UI. 76 minutes.

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

What the test found: jsoup compiles from source with JDK 21 and Apache Maven, all 2,341 unit tests pass, and the built JAR correctly parses HTML and extracts DOM content. 2 minutes.

Elasticsearch File System Crawler (FS Crawler)

What the test found: FSCrawler 3.1-SNAPSHOT builds successfully via Maven, distributable ZIP is packaged, binary launches and shows help/version, and 687 unit tests pass when run with the `-Dtest="*Test"` workaround. 82 minutes.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here...

What the test found: Scrapling v0.4.15 installed and all non-browser tests pass; the core parser, CLI, spider framework, request-based fetchers, and integration with Scrapy all work correctly. 11 minutes.

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

What the test found: Pinchtab builds from source, 53 of 54 Go test packages pass (1 pre-existing handlers failure), and the bridge serves HTTP 200 health checks with Chrome 131 headless. 41 minutes.

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

What the test found: crawler v2.1.1 builds from TypeScript, all 51 tests pass under Node 22 with nock-mocked HTTP, and the module loads correctly to process crawl tasks. 4 minutes.

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

What the test found: All 217 tests pass with 81% coverage; twscrape CLI reports version 0.20.1 and SQLite 3.45.1. 1 minute.

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

What the test found: The Bright Data MCP server installs and all 21 unit tests pass under Node.js v22.12.0; the server starts listening when API_TOKEN is set. 4 minutes.

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

What the test found: Python scanner installs successfully, compiles cleanly, and completes a directory crawl indexing 22 file/directory documents into a mock Elasticsearch instance. 18 minutes.

Finviz analysis python library.

What the test found: All 128 offline tests pass and every package module (quote, news, insider, calendar, screener, group, earnings, forex, crypto, future, util) returns correct data from committed HTML fixtures. 2 minutes.

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.

What the test found: Scweet 5.8.0 installs cleanly in a Python 3.12 venv; 495 unit tests pass; the CLI builds and shows help; the library imports without network; material in tests/fixtures/ provides captured X responses for offline test coverage. 9 minutes.

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24...

What the test found: Fredy backend serves HTTP 200 on port 9998 after DB migration and all 4260 offline tests pass with Node v22.23.3. 6 minutes.

Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

What the test found: Virtual environment created with all dependencies, both Python scripts compile clean, and the full mock-based test suite (109 tests) passes. The CLI prints correct version (2.2.0), help output, and city listings. Real Chrome CDP scraping cannot be verified in this container, --check correctly reports no Chrome... 2 minutes.

Self-hosted SERP API for AI, SEO & automation. Browser-rendered Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia search with page extraction 🎉

What the test found: OpenSERP v0.8.13 builds, all 12 unit test suites pass, and the HTTP server starts, serving /health and /ready endpoints on 127.0.0.1:7000. 8 minutes.

WebAI2API: 基于 Camoufox 的网页 AI 转 API 工具，支持 LMArena/Gemini等，多窗口并发与账号隔离。 | Web AI to OpenAI API via Camoufox. Supports LMArena/Gemini and more, multi-window...

What the test found: The WebAI2API server starts on port 3000 with a Camoufox browser instance, serving OpenAI-compatible /v1/models, WebUI, and /v1/cookies endpoints, all returning HTTP 200, though chat completions require real AI website credentials via the browser adapter and cannot produce output without logging in. 7 minutes.

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude...

What the test found: All 17 Go packages with tests passed with race detection (no failures), the binary compiled and prints version/help correctly, and all 10 JS skill tests plus shell helper tests passed. 6 minutes.

Python scraper based on AI

What the test found: 191 unit tests pass after fixing 13 categories of test failures covering models_tokens assertions, cleanup_html behaviors, logger capture, file paths, missing dependencies, mock patterns, and mock class completeness. 46 minutes.

基于搜狗微信搜索的微信公众号爬虫接口

What the test found: wechatsogou 4.5.4 is installed in a Python 3.12 venv; pip install -e . succeeds; 24 offline tests pass (constants, tools, URL generation, HTML parsing); real-network API tests fail only because DNS is unavailable in the container. 13 minutes.

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG &...

What the test found: Fess 15.9.0-SNAPSHOT builds and packages successfully from source. All 7,634 unit tests pass. The embedded Tomcat binds port 8080 but the application fails at runtime because no OpenSearch backend is reachable on localhost:9200. 36 minutes.

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and...

What the test found: could not be verified; the log shows where it stopped.