Open source web scraping tools, installed and run

Web scraping and crawling tools you can run yourself. The list is ordered by whether each project came up when Argusic installed it fresh, with the recording of that attempt one click away.

Tested between and . Each row shows its own test date; a project can change after that day.

146 of 150 tested projects run. 4 more waiting for a test.

In short: 112 of the 150 tested projects started as-is on a fresh machine: ai-website-cloner-template, colly, katana, Skill_Seekers, newspaper, SeleniumBase, crawlab, and webmagic, and 104 more. 34 more started once a stand-in replaced a service they expect, such as a database: node-crawler, twscrape, brightdata-mcp, boss-zhipin-scraper, Scweet, fredy, x-kit, and chatWeb, and 26 more. 4 could not be verified: WechatSogou, fess, agentql, and Krawl; the log shows where each one stopped.

Measured by Argusic on a fresh machine every time. Every number links to its evidence.

#projectverdictArgusic Scorelanguagestarstested on
1ai-website-cloner-templateRuns100 / 100TypeScript39,625

Clone any website with one command using AI coding agents

What the test found: Next.js 16.3.0 project builds, typechecks, and serves HTTP 200 on the root route via npm run dev. 4 minutes.

2collyRuns100 / 100Go25,609

Elegant Scraper and Crawler Framework for Golang

What the test found: The Colly module builds, vets, passes its full unit and race test suites, its CLI prints help, and its scraper successfully crawled a local mock site end to end. 14 minutes.

3katanaRuns100 / 100Go17,885

A next-generation crawling and spidering framework.

What the test found: katana v1.7.0 is installed, resolves from a fresh login shell, crawls HTTP sites in standard and headless modes, and its unit, functional, and integration suites pass. 7 minutes.

4Skill_SeekersRuns100 / 100Python15,357

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

What the test found: skill-seekers 3.10.0.dev0 installed and all 3857 unit/adaptor/scraper tests pass without external API keys or network dependencies. 6 minutes.

5newspaperRuns100 / 100Python15,192

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

What the test found: newspaper3k 0.3.0 installs in a virtualenv on Python 3.12, all 40 unit tests pass, and core Article download/parse/nlp pipeline produces correct output from static HTML. 4 minutes.

6SeleniumBaseRuns100 / 100Python13,273

Fast and reliable browser automation for e2e testing, scraping, and stealth. Bypasses bot-detection with CDP Mode. Has integrations for pytest, Playwright, and...

What the test found: SeleniumBase 4.54.7 is installed in a Python 3.12 venv; 8 framework unit tests, 7 offline HTML tests, and 2 real web tests passed using headless Chrome 154. 5 minutes.

7crawlabRuns100 / 100Go12,299

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

What the test found: Crawlab v0.6.3 backend compiles, starts as master node, connects to MongoDB, and serves its REST API on port 8000 (system-info, login, nodes, spiders, tasks, users, stats endpoints confirmed reachable). 67 minutes.

8webmagicRuns100 / 100Java11,660

A scalable web crawler framework for Java.

What the test found: The webmagic project compiles and passes all 108 tests with JDK 22 and Maven 3.9.9. Tests exercise real HTTP downloads from github.com (e.g. GithubRepoPageProcessorTest downloads and processes 1 page), SSL compatibility checks, XPath/CSS/regex selectors, file pipeline, proxy providers, duplicate-removal scheduling... 13 minutes.

9crawlee-pythonRuns100 / 100Python9,679

Crawlee, A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the...

What the test found: Crawlee 1.10.3 installs, builds, passes 2480 unit tests, and crawls example.com end to end with both BeautifulSoupCrawler and PlaywrightCrawler after fixing playwright configuration propagation. 35 minutes.

10heliumRuns100 / 100Python8,333

Lighter web automation with Python

What the test found: Helium 7.0.3 installs in a venv and its full 332-test suite passes against a real headless Chrome 157 downloaded by Selenium Manager, with date-input and subprocess tests fixed for this environment. 15 minutes.

11trafilaturaRuns100 / 100Python7,161

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

What the test found: Trafilatura 2.2.0 installed from source in a Python 3.12.3 venv; all 428 tests pass including text extraction, CLI pipeline processing, JSON output, sitemap parsing, feed crawling, URL deduplication, and metadata extraction; the CLI binary and Python API both produce correct output from HTML input. 36 minutes.

12curl_cffiRuns100 / 100Python6,911

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

What the test found: curl_cffi 0.17.0 installed via pre-built wheel, imports work, real HTTP requests succeed with and without impersonation, CLI doctor reports 47 native fingerprints, 711/730 unit tests pass (19 pre-existing skips). 7 minutes.

13ferretRuns100 / 100Go6,025

Declarative data automation language and Go runtime for structured extraction workflows.

What the test found: Ferret v2 repo builds with Go 1.25.0, all 55 test packages pass (unit, integration, security, spec, compat, benchmarks), and the CLI binary correctly evaluates FQL queries (return 1 + 1 → 2). 5 minutes.

14scrapy-redisRuns100 / 100Python5,648

Redis-based components for Scrapy.

What the test found: scrapy-redis 0.9.1 is installed in a venv at /work/repo/.venv with real Redis 6.2 running on localhost:6379; all 102 tests pass covering connection pooling, dupefilter, queue (FIFO/LIFO/priority), scheduler, spider setup, idle backoff, multi-key consumption, and utility modules. 9 minutes.

15scrape-itRuns100 / 100JavaScript4,069

🔮 A Node.js scraper for humans.

What the test found: The scrape-it library installs and all 9 integration tests pass on Node.js v18.19.1 after pinning cheerio to 1.0.0. 2 minutes.

16toapiRuns100 / 100Python3,557

Every web site provides APIs.

What the test found: toapi 2.2.4 installed in editable mode, all 7 pytest tests pass, CLI reports version 2.2.4, ruff linting passes, and the API serves JSON responses via WebTest against example.com. 2 minutes.

17php-curl-classRuns100 / 100PHP3,293

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

What the test found: PHP Curl Class library installed via composer, PHP 8.3.12 with curl 8.10.1, vendor autoload and all 66 dependencies resolved, library classes load correctly (Curl: 152 methods, MultiCurl: 74 methods), core HTTP methods and parallel requests verified working against local test server. 74 minutes.

18google-play-scraperRuns100 / 100JavaScript3,005

Node.js scraper to get data from Google Play

What the test found: The google-play-scraper npm module installs, its index.js exports all 12 documented methods, and all 84 tests pass against live Google Play endpoints. 3 minutes.

19crawlerRuns100 / 100PHP2,827

https://spatie.be/docs/crawler

What the test found: 229 tests pass (552 assertions) for the Spatie Crawler package using a downloaded static PHP 8.4.23 binary. 5 minutes.

20geziyorRuns100 / 100Go2,781

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

What the test found: Every test in the repository passes: go build ./... succeeds silently; go test -count=1 ./... passes all 6 test packages (geziyor, cache, cache/diskcache, cache/leveldbcache, client, export, middleware) with no failures. 5 minutes.

21geccoRuns100 / 100Java2,510

Easy to use lightweight web crawler(易用的轻量化网络爬虫)

What the test found: Gecco 1.3.21 compiles cleanly, packages as a 146K jar, and the framework correctly matches URLs, converts types, builds HTTP requests, registers DynamicGecco beans, and runs a full crawl-pipeline-render cycle against a local HTTP server. 4 minutes.

22Crawler-DetectRuns100 / 100PHP2,410

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

What the test found: PHP 8.3.6 with Phar/iconv/mbstring/readline/xml/dom/curl/tokenizer/xmlwriter extensions is installed via extracted debs, Composer 2.11.2 is installed, project dependencies are resolved, and the test suite passes (28 tests, 170886 assertions). 18 minutes.

23gocloneRuns100 / 100Go2,304

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

What the test found: The goclone website mirroring tool builds, passes all 18 unit tests, and successfully clones real websites (verified with https://example.com). 3 minutes.

24x-crawlRuns100 / 100TypeScript1,888

Flexible Node.js crawler library

What the test found: x-crawl builds successfully and 26 automation tests pass covering crawlData, crawlHTML, crawlPage, async/sync mode, device fingerprint, proxy rotation, and error collection. 9 minutes.

25ruiaRuns100 / 100Python1,738

Async Python 3.6+ web scraping micro-framework based on asyncio

What the test found: Ruia 0.8.5 installed and fully functional inside a venv at /tmp/ruia-venv; all 67 tests pass, package importable as ruia.__version__ == '0.8.5'. 12 minutes.

26selectolaxRuns100 / 100Python1,717

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

What the test found: selectolax 0.4.13 builds from source with Cython and Lexbor, and all 279 tests pass. 7 minutes.

27siteone-crawlerRuns100 / 100Rust1,020

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization, ideal for developers, DevOps...

What the test found: siteone-crawler v2.7.0-dev builds from source, passes all 130 unit tests, and crawls a local page producing valid JSON output. 4 minutes.

28stormcrawlerRuns100 / 100Java1,014

A scalable, mature and versatile web crawler based on Apache Storm

What the test found: Apache StormCrawler 4.0.0-SNAPSHOT builds and all 18 modules compile successfully; 983/983 pass/fail/error (0/0) with 46 Docker-dependent tests skipped; core JAR at core/target/stormcrawler-core-4.0.0-SNAPSHOT.jar (404 KB). 20 minutes.

29x-readerRuns100 / 100Python992

Universal content reader MCP Server for 10+ platforms

What the test found: x-reader installs cleanly in a virtual environment, all 10 unit tests pass, and the CLI fetches real content from URLs via the Jina reader fallback. 2 minutes.

30TikHub-API-Python-SDKRuns100 / 100Python945

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱)...

What the test found: SDK installs, imports, all 110 tests pass, and both sync/async clients successfully reach the real TikHub API health endpoint and the CLI prints version info. 2 minutes.

31icrawlerRuns100 / 100Python937

A multi-thread crawler framework with many builtin image crawlers provided.

What the test found: icrawler 0.1.dev1 installs in Python 3.12 venv, passes 5 test cases (4 real crawler smoke tests + 1 doctest), Bing and Baidu crawlers download valid JPEG/PNG images from the web after fixing 3 runtime bugs. 12 minutes.

32scrapyrtRuns100 / 100Python884

HTTP API for Scrapy spiders

What the test found: ScrapyRT 0.18.1 installed in a venv, all 126 unit and integration tests pass, the HTTP server starts and responds to GET and POST crawl.json endpoints with real Scrapy spider output. 9 minutes.

33scrapecraftRuns100 / 100Python731

🤖 AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph.

What the test found: ScrapeCraft backend serves all API endpoints on uvicorn, WebSocket connections establish and respond, frontend builds successfully, chat uses real OpenRouter API key, JWT auth works, pipeline CRUD works, scraping endpoints respond. 14 minutes.

34ocRuns100 / 100JavaScript694

Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.

What the test found: only-cli v0.5.8 installs and runs on Node 20.11.1; 263 of 264 tests pass; the CLI fetches, distills, and renders public web pages with token-budgeted output and actionable numbered links. 3 minutes.

35NetDiscoveryRuns100 / 100Java643

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

What the test found: The full multi-module Gradle build passes (BUILD SUCCESSFUL, 63 tasks), and the NetDiscovery spider engine example ran as a Java process answering HTTP 200 with JSON on port 8715 routes /netdiscovery/health/, /netdiscovery/spiders/, /netdiscovery/spider/tony1/detail, and /netdiscovery/metrics/, while the downloader... 69 minutes.

36huashu-chromeRuns100 / 100JavaScript610

让任何 AI agent 操控你自己的 Chrome, 带着全部登录态。MCP + Chrome 扩展,Claude Code / Codex / Cursor / Gemini CLI 通用。

What the test found: All 123 unit tests pass, npm install completes, MCP server responds with 22 tools, bridge daemon starts and answers probes, doctor command runs successfully. 2 minutes.

37freeproxyRuns100 / 100Python595

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,每日更新海量免费代理。)

What the test found: pyfreeproxy v0.5.2 installed in /tmp/venv, all 48 proxy sources registered, Proxifly/ProxyScrape/Geonode scrape real proxies successfully, filter rules and ProxiedSessionClient work, JSON export round-trips correctly, free proxies collected but actual proxy requests time out as expected from this container network. 3 minutes.

38social-media-profile-scrapersRuns100 / 100Python588

Fetch user's data across social media

What the test found: 8/8 tests pass; GitHub and Pinterest CLI scrapers fetch real profile data; the 6 remaining scrapers return None gracefully given no browser binary or API access. 16 minutes.

39readerRuns100 / 100TypeScript579

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

What the test found: Reader CLI builds, all 346 unit tests pass, the daemon starts with Playwright's bundled Chromium, and an end-to-end scrape of https://example.com via HTTP returns clean HTML and markdown content with a 200 status. 13 minutes.

40Stealth-RequestsRuns100 / 100Python570

Undetected web-scraping & seamless HTML parsing in Python!

What the test found: stealth-requests 2.1.0 installed in a fresh venv, all 64 tests pass, and real requests to httpbin.org succeed with proper Chrome impersonation headers. 1 minute.

41ticket-monitoringRuns100 / 100Python528

✈️ 自托管的多渠道机票价格监控:航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线(五洲目的地)同一套采集-判定-推送链路;数据留在本机,配置全在网页,开源免费。

What the test found: All dependencies installed in venv, Playwright chromium installed, test suite runs 3712 tests all passing, web UI demo server starts and responds with HTTP 200 on its state API endpoint, main.py --version reports 1.5.303. 44 minutes.

42fundusRuns100 / 100Python480

A very simple news crawler with a funny name

What the test found: Fundus 0.5.7 installs cleanly in a venv, all 1131 unit tests pass, and the crawler successfully fetches live article data from The New Yorker via HTTP. 3 minutes.

43camofox-browserRuns100 / 100JavaScript453

Anti-detection browser server for AI agents, REST API wrapping Camoufox engine with OpenClaw plugin support

What the test found: Camofox Browser Server builds, starts on port 9998, creates browser contexts via Camoufox Firefox headless, navigates to https://example.com, returns accessibility snapshots, and passes the 677-test full suite with Node.js 20 and rebuilt native addons. 34 minutes.

44sparklerRuns100 / 100Python419

A crawl workstation: View, Control, and Crawl. Solr CrawlDB, Tika, Vue 3.

What the test found: Sparkler is installed with Python 3.12, Solr 10.0.0 on port 8983, Vue UI built and served on port 8180, APIs responding, all 32 unit tests passing. 7 minutes.

45HydraRuns100 / 100Java406

为超级个体和一个人公司打造一个人的大厂,Hydra九头龙构筑大规模AI调度、数据采集、情报系统、数据平台、分析决策、产品生产的'军事'工业基座。

What the test found: After fixing module-name casing, pinning Lombok 1.18.34, and adding the missing System.Subsystem config block, the Maven build succeeds and the ShadowBoot crawler app boots and runs the README's Void minimum-system demo to completion (exit 0, 'Pinecone Runtime : 136 /ms !'), while the full test suite passes with 48... 9 minutes.

46trends-checkerRuns100 / 100Python405

Python CLI for Google Trends analysis with advanced rate limiting, cookie auth, and beautiful terminal output. Built on pytrends with enterprise features.

What the test found: trends-checker installs and runs, hitting the real Google Trends API via pytrends (with urllib3 compat patch applied to pytrends request.py), returning real interest data verified at score 42.83 for 'AI agents' in US. 5 minutes.

47ir-searchRuns100 / 100Python394

🇰🇷 한국 정부 지원사업 전수조사 에이전트 스킬 [Claude Code·Codex·agy(Antigravity)·Cursor·Gemini CLI·Grok Build 지원] K-Startup·기업마당·NIPA·KOCCA·SMTECH 크롤링 + 즉시/로드맵/변형 3분류...

What the test found: The project's 229 pytest tests pass, and every CLI crawler (K-Startup, bizinfo, NIPA, KOCCA, SMTECH list and detail, scope_plan, diff_surveys) ran successfully against live Korean government sites, producing real announcement records and detail text. 4 minutes.

48crawlerRuns100 / 100PHP373

Library for Rapid (Web) Crawler and Scraper Development

What the test found: 1278 of 1278 default unit tests pass, PHP 8.3.32 and Composer installed, Chrome for Testing available, integration server fixed with short_open_tag=Off and 500ms startup delay. 27 minutes.

49scrapy-zyte-smartproxyRuns100 / 100Python365

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

What the test found: scrapy-zyte-smartproxy v2.4.1 installed in a Python 3.12 venv; all 82 existing unit tests pass with zero failures. 1 minute.

50hQuery.phpRuns100 / 100PHP358

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

What the test found: hQuery.php library installs, autoloads, and parses HTML via fromHTML() and find()/text() selectors; all 27 unit tests pass on PHP 8.4.1 via FrankenPHP static binary. 7 minutes.

51geo-aeo-trackerRuns100 / 100TypeScript334

Open-source AI visibility tracker and AEO checker. Track how ChatGPT, Perplexity, Gemini and Google AI Mode mention and cite your brand. MCP server + CLI...

What the test found: Next.js dev server runs on port 3000 serving the GEO/AEO Tracker dashboard in demo mode, with all pages (/ and /demo) and API routes (/api/analyze, /api/audit, /api/scrape, etc.) responding correctly. 4 minutes.

52aliexpress-product-scraperRuns100 / 100JavaScript331

Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,

What the test found: Node 24.0.0 installed, pnpm installed, Chromium 145 downloaded, all 22 tests pass, ESLint clean. 1 minute.

53Laravel-Crawler-DetectRuns100 / 100PHP326

A Laravel wrapper for CrawlerDetect - the web crawler detection library

What the test found: All 9 package tests pass, detecting crawlers from real user agent lists via the Laravel facade and Request macros. 4 minutes.

54SpotifyScraperRuns100 / 100Python324

Extract public Spotify data, tracks, albums, artists, playlists, podcasts & lyrics, without the official API. Sync + async, typed models, one dependency.

What the test found: The SpotifyScraper package is installed and working: the CLI fetches real Spotify track/artist/playlist data over the network, the hermetic unit test suite passes completely (921/921), and the MCP server imports and launches correctly after pinning mcp to v1. 3 minutes.

554chan-downloaderRuns100 / 100Python312

Python3 script to continuously download all images/videos of multiple 4chan threads simultaneously - without installation

What the test found: Both scripts compile, install cleanly via venv, and successfully interact with live 4chan API endpoints: thread-watcher queries the catalog and writes URLs, inb4404 downloads thread media files with validated content. 4 minutes.

56xvideosRuns100 / 100TypeScript310

xvideos API library

What the test found: npm install completes with warnings. Build succeeds. 5 unit test files (47 tests) pass on fixture HTML. Full test suite: 52 pass, 4 fail (all integration tests blocked by xvideos.com returning 404 on /verified/videos/). 5 minutes.

57facebook-ads-library-mcpRuns100 / 100Python304

Free, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research, advertiser, copy, landing pages, CTAs, any...

What the test found: MCP server loads, imports, starts over stdio, registers both tools, and all 8 offline parser tests pass. 7 minutes.

58teracrawlRuns100 / 100TypeScript291

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

What the test found: Teracrawl server starts on port 8085 with a local Chromium-backed mock Browser.cash pool and a mock SERP service. GET /health returns 200, POST /scrape converts a live URL to markdown via real headless Chromium, POST /serp/search returns mock search results, and POST /crawl combines both to scrape multiple URLs. 15 minutes.

59OddsHarvesterRuns100 / 100Python276

A python app designed to scrape and process sports betting data directly from oddsportal.com 🎯

What the test found: OddsHarvester v0.15.0 installed from source with uv and Playwright; all 1308 unit tests pass and 63 of 82 integration tests pass against HAR replay (19 live-only skipped, 1 xfail for known in-play replay limitation). 40 minutes.

60go-moviesRuns100 / 100Go275

golang spider Crawler 爬虫 电影

What the test found: GoMovies binary compiled and serves HTTP on port 8899 with Redis persistence; spider successfully fetches movie data from hhzyapi.com external API. 20 minutes.

61Project-Eyes-OnRuns100 / 100Python256

Project Eyes On is a high-speed, multi-threaded surveillance tool by Y0oshi (@rde0) for locating open IP cameras worldwide. Unifies Google Dorking and...

What the test found: Project Eyes On installed, imports correctly, Insecam scraping and multi-engine dorking both find real live IP cameras, and JSON/CSV/HTML reports all produce valid output. 2 minutes.

62Uni-CLIRuns100 / 100TypeScript229

One command for every interface, search, run, and inspect real software across APIs, browsers, desktops, local tools, and MCP.

What the test found: Uni-CLI v1.2.1 builds from source, runs 3735 passing unit tests, and executes live commands (hackernews top, search, doctor) against real remote APIs on Node 22.23.4 (npm v10.9.9). The npm global package v1.3.0 is also installed and functional. 20 minutes.

63patchright-pythonRuns98 / 100Python1,603

Undetected Python version of the Playwright testing and automation library.

What the test found: patchright installed from PyPI, builds from source, and launches patched Chromium headless shell; the package passes 584 of 590 Playwright chromium sync tests with only SSL-cert infrastructure and Xvfb-signal-handling failures. 10 minutes.

64Douyin_TikTok_Download_APIRuns97.3 / 100Python21,518

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video...

What the test found: All 2960 tests pass (2450 unit + 510 integration), the CLI outputs version 5.0.1 and lists 10 commands, the API server starts and serves healthz/readyz/swagger/console root with 200, the web console SPA is built, and both the PostgreSQL+TimescaleDB and Redis backends and the Go downloader sidecar run. 48 minutes.

65camofox-browserRuns96.7 / 100JavaScript12,456

Stealth headless browser for AI agents, bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

What the test found: Camofox browser server installed and verified: Node 22.23.4 in user space, Camoufox 1.3GB binary downloaded, GeoIP 50MB database downloaded, server starts at localhost:9377 with Xvfb virtual display, creates browser tabs navigating to real URLs, returns accessibility snapshots with element refs, and closes cleanly on... 13 minutes.

66browserlessRuns96.7 / 100JavaScript1,846

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

What the test found: All 10 of 14 testable packages pass fully (330 tests) on Node v24.0.0 with Puppeteer 25.17.0 and Chromium 157.0.8103.50; the sole regression is the navigator.languages evasion returning an extra language tag due to the newer Chrome version. 26 minutes.

67ScraplingRuns96 / 100Python104,979

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ...

What the test found: Scrapling 0.4.15 is installed in a venv with all extras, Playwright Chromium browser available, all 985 tests pass, CLI and MCP server commands respond correctly, and HTTP fetches via StealthyFetcher return status 200 from real endpoints. 8 minutes.

68proxy_poolRuns96 / 100Python23,800

Python ProxyPool for web spider

What the test found: The ProxyPool Flask API server runs on port 5010 backed by real Redis, responds to all endpoints (/get, /pop, /all, /delete, /count), and all 248 automated tests pass. 6 minutes.

69NoriaRuns96 / 100JavaScript4,623

An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices...

What the test found: All 232 tests pass across 17 suites; the Node.js pipeline orchestrator boots, loads pipelines, and displays the WhatsApp QR code; the Streamlit Control Center starts on :8501; ESLint passes; Prettier reports 2 non-critical formatting warnings in src/infrastructure/messaging/channel-fetcher.js and src/launcher.js. 3 minutes.

70reverse-api-engineerRuns96 / 100Python4,175

Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor and OpenCode.

What the test found: All dependencies install cleanly via pip, the CLI loads and responds to --version/--help/list--json, and the full pytest suite (1031 tests) passes. 7 minutes.

71owllookRuns96 / 100Python2,900

owllook-小说搜索引擎

What the test found: The owllook web server launches on port 8001 and responds with 200 HTTP status on /, /novels/search, /novels/top, /api/search, and /health routes, with real HTML content, connecting to running Redis and MongoDB instances. The existing pytest test suite passes (1 passed, 0 failed). 15 minutes.

72diskover-communityRuns96 / 100PHP1,831

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

What the test found: Elasticsearch 7.13.1 cluster green on localhost:9200, Python 3.12 virtualenv with diskover scanner indexes files to ES, and PHP 8.2.32 built-in server serves the diskover-web UI authenticating with default credentials diskover/darkdata showing both indexed indices. 23 minutes.

73finvizfinanceRuns96 / 100Python1,790

Finviz analysis python library.

What the test found: finvizfinance v1.5.0 installed and builds. Quote, calendar, insider, news, group overview, crypto, forex, and futures all verified working against live finviz.com. Screener and earnings remain broken due to finviz SPA migration requiring a larger rewrite. 13 minutes.

74openserpRuns96 / 100Go1,608

Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page...

What the test found: OpenSERP builds from source, all 12 unit test suites pass, the HTTP server serves /health (200) and handles live DuckDuckGo searches (200) returning structured JSON with organic results and SERP features. 6 minutes.

75WebAI2APIRuns96 / 100JavaScript1,428

WebAI2API: 基于 Camoufox 的网页 AI 转 API 工具,支持 LMArena/Gemini等,多窗口并发与账号隔离。 | Web AI to OpenAI API via Camoufox. Supports LMArena/Gemini and more, multi-window...

What the test found: The server starts, launches a Camoufox browser instance in headful mode on Xvfb, serves the OpenAI-compatible API on port 3000 returning 200 for /v1/models and WebUI, the SQLite history database initializes, all 19 adapters load, and the browser fingerprint is generated and saved. 6 minutes.

76httpcloakRuns96 / 100Go1,388

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic...

What the test found: The httpcloak Go library builds and all 8 test packages pass cleanly. Example programs compile without vet warnings. 7 minutes.

77wombatRuns96 / 100Ruby1,360

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

What the test found: Wombat 3.3.0 builds and installs as a gem under Ruby 3.2.3, all 56 RSpec tests pass, and a live Wombat.crawl scrape of a local WEBrick page returns correctly parsed structured data. 12 minutes.

78newspaper4kRuns96 / 100Python1,170

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

What the test found: newspaper4k 0.9.6 is pip-installed and editable in /work/repo/venv; CLI runs (help + real article extraction into JSON), 234 tests pass with 2 deselected stale summarization fixtures and 5 external-site failures; package state is clean working tree. 38 minutes.

79parse-videoRuns96 / 100Go1,104

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

What the test found: The project builds, all tests pass (0 cmd tests, 0 parser tests with zero failures), the serve subcommand starts and responds on HTTP, the version subcommand prints 'parse-video dev', and the parse API returns correct error responses. 3 minutes.

80seonautRuns96 / 100Go873

Open source SEO audit tool.

What the test found: SEOnaut installed from source (Go build OK, frontend CSS built, migrations applied to a local MariaDB) and ran on port 9000, serving sign-in (200), creating a user and project, completing a real crawl of 2 URLs, and rendering the dashboard. 9 minutes.

81primpRuns96 / 100Rust635

HTTP client that can impersonate web browsers

What the test found: The repo builds and runs: Rust workspace checks and per-crate test suites pass (1354 tests), the primp Python extension installs into crates/primp-python/.venv, 493 Python tests pass, and a real impersonated HTTPS request returns HTTP 200. 15 minutes.

82xxl-crawlerRuns95.7 / 100Java756

A lightweight web crawler framework.(Java爬虫框架)

What the test found: XXL-CRAWLER builds and packages successfully; all 9 runnable tests pass including 4 real internet crawls against cnblogs.com, baidu.com, and 163.com with live data returned. 8 minutes.

83patchright-nodejsRuns95 / 100TypeScript813

Undetected NodeJS version of the Playwright testing and automation library.

What the test found: npm install and tsc --noEmit pass; patchright_rebranding.ts transforms Playwright 1.65.1 into Patchright npm packages; the patched CLI returns version 1.65.1 and chromium browser can be installed and launched headlessly via the patched API, navigating to pages and evaluating JavaScript correctly. 9 minutes.

84pinchtabRuns94.3 / 100Go10,603

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

What the test found: A source-built PinchTab server (binary reports pinchtab dev) is running on 127.0.0.1:9867 with a responsive always-on headless instance and answered authenticated health, navigate, snapshot, text, title, and screenshot API calls with HTTP 200, while the full Go unit suite (9166 passed, 18 skipped) and scripts/check.sh... 19 minutes.

85BeanbunRuns93.5 / 100PHP1,258

Beanbun 是用 PHP 编写的多进程网络爬虫框架,具有良好的开放性、高可扩展性,基于 Workerman。

What the test found: Beanbun 1.0.4 with vendored workerman and guzzle dependencies. PHP 8.3.6 extracted from apt. All source and vendor files parse without errors. Framework class instantiation, configuration, and all Helper methods (randUserAgent, formatUrl, getUrlByHtml) verified working. 16 minutes.

86deepcrawlRuns93.3 / 100TypeScript704

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

What the test found: Dependencies installed, SDK tests pass (41/41), full monorepo build and typecheck pass, and the dashboard serves HTTP 200 on / and /docs in both dev and production modes; the V0 API worker served 200 on /, /docs and /openapi in local mode with D1 migrations applied. 33 minutes.

87rodRuns91.3 / 100Go7,161

A Chrome DevTools Protocol driver for web automation and scraping.

What the test found: Rod project builds, rod-manager binary runs, and full test suite executes against real Chromium 128.0.6568.0 with 211/217 passing. 65 minutes.

88nanobrowserRuns90.7 / 100TypeScript14,368

Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.

What the test found: Dependencies installed, project builds to dist/ (background.iife.js 1,848.81 kB, manifest.json), TypeScript type-check passes across all 15 packages, unit tests pass (14/14). 15 minutes.

89firecrawl-mcp-serverRuns90.2 / 100JavaScript7,884

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any...

What the test found: Firecrawl MCP Server v3.24.1 builds, installs, and starts successfully with 25 tools in stdio mode; 81 of 84 tests pass. 24 minutes.

90scrapyRuns90 / 100Python65,539

Scrapy, a fast high-level web crawling & scraping framework for Python.

What the test found: Scrapy 2.19.0 installed from source, 4539 tests pass, scrapy bench crawls 1781 pages against a local HTTP server. 15 minutes.

91luxRuns90 / 100Go31,855

👾 Fast and simple video download library and CLI tool written in Go

What the test found: Lux builds and runs: the binary outputs version info and help, extracts real YouTube video metadata, and 14 of 48 test packages pass (all failures are network-dependent extractor tests or DNS-timeout content-type lookups; no code defects remain). 11 minutes.

92news-pleaseRuns90 / 100Python2,491

news-please - an integrated web crawler and information extractor for news that just works

What the test found: news-please 1.6.13 installs and its library API extracts structured article data (title, authors, publication date, description, maintext, language) from HTML via both HTTP and raw HTML input; CLI entry points respond with help output. 5 minutes.

93top-user-agentsRuns90 / 100JavaScript385

Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.

What the test found: The repo installs cleanly; with the bundled prebuilt Node 22, npm test passes all 5 tests and require/consumer checks return 100 main, 82 desktop, and 18 mobile valid user-agents from the packed tarball; only the Redis-backed update scripts remain unrunnable in this container because no redis-server is available. 3 minutes.

94skillsRuns85 / 100Python6,417

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution...

What the test found: browser-act-cli 1.4.2 installed, skill handshake completed, and the chrome browser type (bundled Chromium) works end-to-end, browser create, open, navigate, state inspection, content extraction via get markdown, and session close all succeed; stealth features require a server-side-validated BrowserAct API key that... 5 minutes.

95nebulaRuns82.5 / 100Go380

🌌 An agnostic network crawler exposing comprehensive peer information and network topology information.

What the test found: Nebula builds with just build, runs dry-run and ClickHouse-backed IPFS crawls against real network peers, persists crawl data to ClickHouse, and passes all 92 tests across 7 packages with Postgres 16.4 and ClickHouse 26.11 running locally on their test ports. 16 minutes.

96browser-useRuns66.7 / 100Python121,752

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

What the test found: browser-use v0.13.10 installed in a Python 3.12 venv with uv, Chromium headless installed via playwright, and all 563 CI tests pass (529 passed, 34 skipped) after fixing 2 test identity comparison bugs in test_beta_agent.py that only manifest under xdist worker parallelism. 75 minutes.

97SurfSenseRuns66.3 / 100Python16,927

Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord...

What the test found: All Python trees (surfsense_backend, surfsense_mcp, surfsense_local/backend, plugins/remote/host core tests) install and pass tests; the web frontend fails to compile native deps on Node 18.x (requires 24.x); docs are clean. 20 minutes.

98changedetection.ioRuns60 / 100Python35,963

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock...

What the test found: changedetection.io 0.60.7 installs into a venv, launches and serves HTTP 200, its full basic test suite passes (680 passed in the main group, exit 0, plus 508 unit tests), and a real watch added via the API detected a content change and exposed it in the UI. 76 minutes.

99jsoupRuns60 / 100Java11,418

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

What the test found: jsoup compiles from source with JDK 21 and Apache Maven, all 2,341 unit tests pass, and the built JAR correctly parses HTML and extracts DOM content. 2 minutes.

100JMComic-Crawler-PythonRuns60 / 100Python8,165

Python API for JMComic | 提供Python API访问禁漫天堂,同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

What the test found: Package jmcomic 2.7.7 installed in a Python 3.12.3 venv with all dependencies; 238/240 tests pass including real network calls against JMComic API servers; CLI commands jmcomic and jmv produce correct help output. 10 minutes.

101stealth-browser-mcpRuns60 / 100Python3,385

Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.

What the test found: Stealth Browser MCP server runs on stdio transport, spawns a headless Chromium 157.0.8086.0, navigates to data URLs, extracts page HTML content, and closes cleanly -- all via MCP protocol over stdio. 15 minutes.

102spiderRuns60 / 100Rust2,857

Foundational low latency web data collecting in Rust

What the test found: Rust 1.99.0 installed, cargo build succeeds, spider CLI binary crawls and scrapes example.com returning status 200 and expected page content, all 608 spider core tests pass, workspace-wide crate tests all pass. 56 minutes.

103google-maps-scraper-kitRuns60 / 100Python2,581

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and...

What the test found: The scraper binary runs natively, its REST API on localhost:8080 creates/polls/downloads scraping jobs against real Google Maps data, and the included Python CLI successfully produces lead CSVs with real business listings. 7 minutes.

104gainRuns60 / 100Python2,017

Web crawling framework based on asyncio.

What the test found: All 7 tests pass, ruff lint passes, library imports correctly, and item attribute extraction from HTML produces expected results. 1 minute.

105wreq-pythonRuns60 / 100Python1,529

An ergonomic, privacy-aware Python HTTP Client

What the test found: wreq Python package builds, installs, and passes all 57 unit tests against a local httpbin server, supporting HTTP/1.1 and HTTP/2 requests, cookie jars, redirect policies, TLS emulation (JA3/JA4), multipart uploads, streaming, and compression. 15 minutes.

106rebrowser-patchesRuns60 / 100JavaScript1,471

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to...

What the test found: rebrowser-patches repo installs, builds 6 drop-in tarballs, and passes all 43 test gates (detection, scenarios, smoke) with patched Puppeteer-core 25.14.0 and Playwright-core 1.64.1 defeating Runtime.Enable detection. 8 minutes.

107fscrawlerRuns60 / 100Java1,459

Elasticsearch File System Crawler (FS Crawler)

What the test found: FSCrawler 3.1-SNAPSHOT builds successfully via Maven, distributable ZIP is packaged, binary launches and shows help/version, and 687 unit tests pass when run with the `-Dtest="*Test"` workaround. 82 minutes.

108crawlyRuns60 / 100Elixir1,129

Crawly, a high-level web crawling & scraping framework for Elixir.

What the test found: Crawly 0.17.2 Elixir library compiles, starts its Cowboy HTTP API on port 4001 answering 200, and runs its 136-test suite with 0 failures and 1 excluded test. 11 minutes.

109wreqRuns60 / 100Rust1,121

An ergonomic, privacy-aware Rust HTTP Client

What the test found: wreq v0.16.1 builds and all 280+ integration tests pass on Rust 1.99.0 (Ubuntu 24.04) with BoringSSL via btls, after providing libclang (extracted from libclang1-20 + libclang-20-dev debs) and workaround for Ubuntu 24.04's missing /usr/include/stddef.h. 35 minutes.

110jvppeteerRuns60 / 100Java805

Java API For Chrome and Firefox

What the test found: Jvppeteer 3.6.5 builds and installs via Maven, Chrome for Testing 149.0.7798.0 runs headlessly, and the library launches the browser, navigates pages, evaluates JavaScript, and retrieves page content successfully. 26 minutes.

111agent-fetchRuns60 / 100TypeScript316

Full-content web fetcher for AI agents, Chrome TLS fingerprinting, browser impersonation, multi-strategy article extraction, and web scraping.

What the test found: agent-fetch 0.1.6 builds from source, passes all 904 unit tests, and successfully fetches and extracts real web pages (Wikipedia, example.com) via CLI. 6 minutes.

112crawl4aiRuns45.7 / 100Python87,664

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this...

What the test found: Crawl4AI 0.9.4 installed in venv, Playwright chromium browser downloaded, DB initialized, AsyncWebCrawler can crawl real URLs with markdown output, cache operations work, and 360 core tests pass with 11 skipped. 14 minutes.

113node-crawlerRuns with mocks92 / 100TypeScript6,797

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

What the test found: installed deps with local Node v22.14.0, built dist, and the full test suite passes (51/51) with HTTP/2 tests served by a local TLS HTTP/2 mock after nghttp2.org returned 502. 54 minutes.

114twscrapeRuns with mocks92 / 100Python2,985

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

What the test found: twscrape 0.20.1 is installed in a Python venv, its CLI answers the version command (twscrape: 0.20.1, SQLite runtime 3.45.1), and its mock-backed pytest suite passes all 217 tests. 2 minutes.

115brightdata-mcpRuns with mocks92 / 100JavaScript2,733

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

What the test found: All 21 unit tests pass with Node 20.20.2. The MCP server loads fastmcp without errors and rejects missing API_TOKEN. Dependencies installed cleanly. 2 minutes.

116boss-zhipin-scraperRuns with mocks92 / 100Python1,784

Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

What the test found: All 109 mocked tests pass, both scripts compile, CLI help and city listing work, version is consistent across all 5 files, and the environment check correctly reports the absence of a real Chrome CDP endpoint. 1 minute.

117ScweetRuns with mocks92 / 100Python1,667

Twitter scraper and X scraper for Python: scrape tweets, profiles, followers, following and trends without the official API. Twitter scraping library with...

What the test found: Scweet 5.8.0 installed from source, all 495 unit tests pass, CLI help works, package imports correctly. The library requires real X accounts and auth tokens for live scraping. 12 minutes.

118fredyRuns with mocks92 / 100JavaScript1,650

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24...

What the test found: Fredy backend serves UI on port 9998 with all 45 migrations applied, login authenticates admin/admin, and all 4260 offline tests pass. 5 minutes.

119x-kitRuns with mocks92 / 100Python947

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

What the test found: uv sync installs 14 packages, all 7 main scripts import without error, 5 of 7 scripts run against a local mock X.com API server returning realistic GraphQL payloads; real X.com credentials (auth_token + ct0) needed for live production use. 8 minutes.

120chatWebRuns with mocks92 / 100Python916

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

What the test found: ChatWeb launches in both API mode (FastAPI/uvicorn on :9531) and WebUI mode (Gradio on :7860). File upload, summary generation, and Q&A endpoints all return HTTP 200 with correct JSON schema against a mock OpenAI API. The three startup modes (console/api/webui) are all functional; console mode was not interactively... 9 minutes.

121ScavengerRuns with mocks92 / 100Python884

Crawler (Bot) searching for leaks on paste sites.

What the test found: The app installs cleanly, all 6 Python modules compile without errors, the core credential detection correctly identifies email:password pairs and sensitive keyword matches via configurable search terms and regex patterns, the SQLite paste-ID dedup tracker and file archiver both function correctly, and the dashboard... 5 minutes.

122spidrRuns with mocks92 / 100Ruby835

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

What the test found: Spidr 0.7.2 on Ruby 3.3.6: 317 RSpec tests pass with 0 failures, gem dependencies installed, web spidering via WebMock-stubbed Sinatra apps works correctly. 33 minutes.

123everyinfra-agent-pluginsRuns with mocks92 / 100Python697

EveryInfra MCP server & agent plugin: scraper API for TikTok, Instagram, Amazon, YouTube, Xiaohongshu & Douyin. 612 data APIs across 106 platforms, from...

What the test found: EveryInfra plugin manifests, skills, MCP configs, marketplace sources and build scripts validate cleanly; deterministic release archive and 13 host-package ZIPs build successfully; the example proxy+data API flow runs correctly against local mocks. 3 minutes.

124catbusRuns with mocks92 / 100TypeScript642

一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用

What the test found: catbus CLI is installed, builds, and all 962 offline tests pass. The binary outputs valid JSON envelopes for version, help, platforms, doctor, and handles platform aliases, native name mapping, and auth-required errors correctly. 3 minutes.

125browser-searchRuns with mocks92 / 100JavaScript550

A skill for AI agents: search the web with SearXNG, browse with Camofox, bypass protections with CloakBrowser. Anti-hallucination by design. Self-hosted, free...

What the test found: npm install succeeds, CloakBrowser (with Node 20, not the system Node 18) fetches live pages, the SearXNG and Camofox deterministic scripts pass their full command flows against schema-correct mocks, unit suites pass 28/28 and 12/12, and setup.sh/audit.sh complete without failed checks; real SearXNG/Camofox Docker... 33 minutes.

126scrappleRuns with mocks92 / 100Python506

A framework for creating semi-automatic web content extractors

What the test found: Scrapple 0.3.0 installs, builds, and runs on Python 3.12; the CLI accepts all 4 commands (genconfig, run, generate, web), scrapes live websites via XPath and CSS selectors, and generates executable Python scripts, 13/17 tests pass with the remaining 4 failing only because their target websites changed HTML since 2018. 10 minutes.

127wreq-jsRuns with mocks92 / 100TypeScript464

HTTP client for Node.js with browser TLS fingerprint impersonation

What the test found: wreq-js v3.2.0 is installed and working on x86_64 Ubuntu 24.04 with Node.js v22.23.4, the TypeScript build completes, the native Rust binding (prebuilt) loads correctly, and the test suite passes 195/200 tests against a local HTTP/HTTPS/WebSocket test server. 5 tests are skipped: 4 requiring full native setCookies... 28 minutes.

128email-sleuthRuns with mocks92 / 100Rust430

Discover and verify professional emails using names + domains

What the test found: Project builds from source, 9 unit tests pass, and both email-sleuth/es binaries execute their full pipeline (config loading, SMTP connectivity testing, pattern generation, DNS resolution, JSON output) without crashing. 6 minutes.

129google-news-scraperRuns with mocks92 / 100JavaScript384

Lightweight scraper for Google News

What the test found: TypeScript compiles cleanly, rollup builds CJS+ESM bundles, both module formats load at runtime, and all 46 unit/integration tests pass on Node 18.19.1. 13 minutes.

130news-crawlRuns with mocks92 / 100Java384

News crawling with StormCrawler - stores content as WARC

What the test found: The NewsCrawlBootstrap topology runs in Storm LocalCluster against local mocks, detects the RSS feed, acks status updates to OpenSearch, and writes a gzip-valid WARC containing fetched feed content; mvn clean package and all 10 unit tests pass. 69 minutes.

131graphlit-mcp-serverRuns with mocks92 / 100TypeScript379

Model Context Protocol (MCP) Server for Graphlit Platform

What the test found: Graphlit MCP Server builds, launches, and responds correctly to MCP protocol initialize, resources/list, and tools/list requests. 10 minutes.

132twitter-scraper-seleniumRuns with mocks92 / 100Python344

Python's package to scrap Twitter's front-end easily

What the test found: Package installs in a virtualenv, all 9 tests pass, xquik search function verified end-to-end with a mock HTTP server. 2 minutes.

133gplay-scraperRuns with mocks92 / 100Python315

GPlay Scraper is a powerful Python Google Play scraper library for extracting comprehensive app data from the Google Play Store. Scrape Google Play Store apps...

What the test found: gplay-scraper installed in a venv with all dependencies; all 48 tests pass after fixing the tls_client client_identifier string from 'chrome112' to 'chrome_112'. 4 minutes.

134MinerU-HTMLRuns with mocks92 / 100Python302

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data...

What the test found: MinerU-HTML v1.1.2 package installs and imports correctly in a Python 3.12 venv; the full extraction pipeline runs to completion with a mock LLM backend, correctly filtering navigation/auxiliary elements from main content HTML; all 20 internal modules import cleanly; config validation, fallback handlers... 16 minutes.

135laravel-seo-scannerRuns with mocks92 / 100PHP275

Scan your Laravel application routes for SEO improvements suggestions.

What the test found: composer install works, the full Pest suite passes (171 passed, 3 skipped), PHPStan passes, and seo:scan-url returns real results (score 65, 28 passed, 15 failed) plus seo:scan persists the score to MariaDB, all against local mocks in this container. 28 minutes.

136examplesRuns with mocks92 / 100TypeScript241

Examples for using Hyperbrowser

What the test found: After npm install, TypeScript compiles cleanly and the company-researcher CLI runs end-to-end against a schema-correct local Hyperbrowser mock, producing the full structured research output and exiting 0; real Hyperbrowser/OpenAI API keys were not exercised. 12 minutes.

137google-maps-scraperRuns with mocks86 / 100Go8,319

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude...

What the test found: All 17 Go packages with tests passed with race detection (no failures), the binary compiled and prints version/help correctly, and all 10 JS skill tests plus shell helper tests passed. 6 minutes.

138awesome-agent-apisRuns with mocks86 / 100Python1,070

660+ muapi-hosted generative-media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping), one YAML file per entry...

What the test found: PyYAML 6.0.3 installed; catalog_validate.py validates all 665 YAML entries in providers/ and models/ successfully with zero errors. 3 minutes.

139extractorRuns with mocks72 / 100TypeScript322

Use LLMs to robustly extract web data

What the test found: npm install builds successfully, 84 unit tests pass, 63 html-to-markdown integration tests pass with real ground-truth data, all LLM extraction integration tests pass with mocked providers, processedContent tests pass; Playwright browser tests fail at import on Node 18. 39 minutes.

140Scrapegraph-aiRuns with mocks56 / 100Python32,890

Python scraper based on AI

What the test found: 191 unit tests pass after fixing 13 categories of test failures covering models_tokens assertions, cleanup_html behaviors, logger capture, file paths, missing dependencies, mock patterns, and mock class completeness. 46 minutes.

141DecodoRuns with mocks56 / 100Java1,264

HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.

What the test found: Mock proxy verified: shell/shell.sh (curl via proxy): returned 200 with IP JSON; python/python.py (requests via proxy): returned 200 with IP JSON; nodejs/node.js (axios via proxy): returned 200 with IP JSON. 9 minutes.

142gpt-promo-scannerRuns with mocks56 / 100Python948

ChatGPT Team(Business) 促销码自动扫描工具, 批量发现/验证/价格收集,支持 17 国 34 个码,最高折扣 71% | ChatGPT Business promo code scanner, batch discovery, validation, price collection, 34...

What the test found: Repo installs cleanly and every scanning script (auto_scan, discover_codes, verify, open_stripe, partner_scan, and the us/mega/uk scanners) ran and produced valid output against the local mock of the ChatGPT API and Clash proxy; open_stripe.py works after switching its hardcoded proxy to the project config-driven... 43 minutes.

143agent-skillsRuns with mocks56 / 100JavaScript869

Official Agent skills of Oxylabs products

What the test found: Five Oxylabs agent skills (headless-browser, proxies, video-data, web-scraper-api, web-unblocker) install via skills CLI. .env.example corrected to OXY_RES_USERNAME/OXY_RES_PASSWORD consistent with all docs. Mock server confirms the documented API endpoint patterns respond 200 OK. 7 minutes.

144google-search-results-pythonRuns with mocks56 / 100Python762

Google Search Results via SERP API pip Python Package

What the test found: The SerpApi Python client library installs from source, runs 43 unit tests successfully against a locally hosted mock backend, and passes a basic smoke test (OOBT). 11 minutes.

145google-reviews-scraper-proRuns with mocks56 / 100Python350

A powerful Google Maps review scraper that works in 2026. Extracts multi-language reviews with images, handles MongoDB integration, and bypasses detection...

What the test found: Python venv setup complete, all dependencies installed, 449/462 tests pass, API server runs and responds 200 on health and data endpoints, all CLI commands operational. 17 minutes.

146quantumproxies.ioRuns with mocks51 / 100Java186

Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries, plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#...

What the test found: Python and Node.js quickstarts and curl HTTP-examples work through mock proxy to real ipinfo.io; PHP/Go/Ruby/Java/C# examples not tested due to missing runtimes; SOCKS5 not tested due to minimal mock. 5 minutes.

147WechatSogouCould not verify80 / 100Python6,411

基于搜狗微信搜索的微信公众号爬虫接口

What the test found: wechatsogou 4.5.4 is installed in a Python 3.12 venv; pip install -e . succeeds; 24 offline tests pass (constants, tools, URL generation, HTML parsing); real-network API tests fail only because DNS is unavailable in the container. 13 minutes.

148fessCould not verify47.5 / 100Java1,152

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG &...

What the test found: Fess 15.9.0-SNAPSHOT builds and packages successfully from source. All 7,634 unit tests pass. The embedded Tomcat binds port 8080 but the application fails at runtime because no OpenSearch backend is reachable on localhost:9200. 36 minutes.

149agentqlCould not verify20 / 100Python1,510

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and...

What the test found: could not be verified; the log shows where it stopped.

150KrawlCould not verify10 / 100Python844

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities...

What the test found: Krawl standalone mode builds and runs: after pip install -r requirements.txt, uvicorn app:app --app-dir src serves HTTP 200 on /dashf/healthz, answers the honeypot with 204 on /, records POST credentials from 198.51.100.77 into SQLite, and the dashboard API shows that IP categorized as attacker. 92 minutes.

-moliNot yet tested-Rust24,245-
-scyllaNot yet tested-Python4,021-
-ketchNot yet tested-Go756-
-tiktok-web-signatureNot yet tested-Python445-

runs installed and started with its real dependencies. runs with mocks started after stand-ins replaced external services such as a database or a third-party API. could not verify neither the standard agent nor the stronger one got it running within the time limit; the log shows where it stopped.

How we tested

On this list as of the latest test: 112 projects ran as-is, 34 with mocks, 4 could not be verified, 4 still waiting. Languages tested: Elixir, Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript. Every attempt used a clean single-use machine, the subject at a pinned version, and a 90-minute limit for standard projects; the complete procedure is on the methodology page.

Frequently asked questions (FAQs)

How is this list ranked?

By measurement, not opinion: projects Argusic installed and launched on a fresh machine come first, then those that ran with mocks in place of external services, then those it could not verify. Ties go to the Argusic Score, then how popular it is on its own source.

Why are some projects unranked?

4 projects are still waiting for a test or for a finished attempt. They are listed without a rank until Argusic has measured them.

Where is the evidence?

Every row links to the project's Argusic page, where each run has a full log and a terminal recording stored with a sha256 fingerprint. The same pages exist for every one of the tested projects, on this list or not.

More lists in this category

All lists: Best. All tested projects: subjects.