Open source web scraping tools, installed and run
Web scraping and crawling tools you can run yourself. The list is ordered by whether each project came up when Argusic installed it fresh, with the recording of that attempt one click away.
Tested between and . Each row shows its own test date; a project can change after that day.
146 of 150 tested projects run. 4 more waiting for a test.
In short: 112 of the 150 tested projects started as-is on a fresh machine: ai-website-cloner-template, colly, katana, Skill_Seekers, newspaper, SeleniumBase, crawlab, and webmagic, and 104 more. 34 more started once a stand-in replaced a service they expect, such as a database: node-crawler, twscrape, brightdata-mcp, boss-zhipin-scraper, Scweet, fredy, x-kit, and chatWeb, and 26 more. 4 could not be verified: WechatSogou, fess, agentql, and Krawl; the log shows where each one stopped.
Measured by Argusic on a fresh machine every time. Every number links to its evidence.
| # | project | verdict | Argusic Score | language | stars | tested on |
|---|---|---|---|---|---|---|
| 1 | ai-website-cloner-template | Runs | 100 / 100 | TypeScript | 39,625 | |
Clone any website with one command using AI coding agents What the test found: Next.js 16.3.0 project builds, typechecks, and serves HTTP 200 on the root route via npm run dev. 4 minutes. | ||||||
| 2 | colly | Runs | 100 / 100 | Go | 25,609 | |
Elegant Scraper and Crawler Framework for Golang What the test found: The Colly module builds, vets, passes its full unit and race test suites, its CLI prints help, and its scraper successfully crawled a local mock site end to end. 14 minutes. | ||||||
| 3 | katana | Runs | 100 / 100 | Go | 17,885 | |
A next-generation crawling and spidering framework. What the test found: katana v1.7.0 is installed, resolves from a fresh login shell, crawls HTTP sites in standard and headless modes, and its unit, functional, and integration suites pass. 7 minutes. | ||||||
| 4 | Skill_Seekers | Runs | 100 / 100 | Python | 15,357 | |
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection What the test found: skill-seekers 3.10.0.dev0 installed and all 3857 unit/adaptor/scraper tests pass without external API keys or network dependencies. 6 minutes. | ||||||
| 5 | newspaper | Runs | 100 / 100 | Python | 15,192 | |
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs: What the test found: newspaper3k 0.3.0 installs in a virtualenv on Python 3.12, all 40 unit tests pass, and core Article download/parse/nlp pipeline produces correct output from static HTML. 4 minutes. | ||||||
| 6 | SeleniumBase | Runs | 100 / 100 | Python | 13,273 | |
Fast and reliable browser automation for e2e testing, scraping, and stealth. Bypasses bot-detection with CDP Mode. Has integrations for pytest, Playwright, and... What the test found: SeleniumBase 4.54.7 is installed in a Python 3.12 venv; 8 framework unit tests, 7 offline HTML tests, and 2 real web tests passed using headless Chrome 154. 5 minutes. | ||||||
| 7 | crawlab | Runs | 100 / 100 | Go | 12,299 | |
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架 What the test found: Crawlab v0.6.3 backend compiles, starts as master node, connects to MongoDB, and serves its REST API on port 8000 (system-info, login, nodes, spiders, tasks, users, stats endpoints confirmed reachable). 67 minutes. | ||||||
| 8 | webmagic | Runs | 100 / 100 | Java | 11,660 | |
A scalable web crawler framework for Java. What the test found: The webmagic project compiles and passes all 108 tests with JDK 22 and Maven 3.9.9. Tests exercise real HTTP downloads from github.com (e.g. GithubRepoPageProcessorTest downloads and processes 1 page), SSL compatibility checks, XPath/CSS/regex selectors, file pipeline, proxy providers, duplicate-removal scheduling... 13 minutes. | ||||||
| 9 | crawlee-python | Runs | 100 / 100 | Python | 9,679 | |
Crawlee, A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the... What the test found: Crawlee 1.10.3 installs, builds, passes 2480 unit tests, and crawls example.com end to end with both BeautifulSoupCrawler and PlaywrightCrawler after fixing playwright configuration propagation. 35 minutes. | ||||||
| 10 | helium | Runs | 100 / 100 | Python | 8,333 | |
Lighter web automation with Python What the test found: Helium 7.0.3 installs in a venv and its full 332-test suite passes against a real headless Chrome 157 downloaded by Selenium Manager, with date-input and subprocess tests fixed for this environment. 15 minutes. | ||||||
| 11 | trafilatura | Runs | 100 / 100 | Python | 7,161 | |
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML What the test found: Trafilatura 2.2.0 installed from source in a Python 3.12.3 venv; all 428 tests pass including text extraction, CLI pipeline processing, JSON output, sitemap parsing, feed crawling, URL deduplication, and metadata extraction; the CLI binary and Python API both produce correct output from HTML input. 36 minutes. | ||||||
| 12 | curl_cffi | Runs | 100 / 100 | Python | 6,911 | |
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints. What the test found: curl_cffi 0.17.0 installed via pre-built wheel, imports work, real HTTP requests succeed with and without impersonation, CLI doctor reports 47 native fingerprints, 711/730 unit tests pass (19 pre-existing skips). 7 minutes. | ||||||
| 13 | ferret | Runs | 100 / 100 | Go | 6,025 | |
Declarative data automation language and Go runtime for structured extraction workflows. What the test found: Ferret v2 repo builds with Go 1.25.0, all 55 test packages pass (unit, integration, security, spec, compat, benchmarks), and the CLI binary correctly evaluates FQL queries (return 1 + 1 → 2). 5 minutes. | ||||||
| 14 | scrapy-redis | Runs | 100 / 100 | Python | 5,648 | |
Redis-based components for Scrapy. What the test found: scrapy-redis 0.9.1 is installed in a venv at /work/repo/.venv with real Redis 6.2 running on localhost:6379; all 102 tests pass covering connection pooling, dupefilter, queue (FIFO/LIFO/priority), scheduler, spider setup, idle backoff, multi-key consumption, and utility modules. 9 minutes. | ||||||
| 15 | scrape-it | Runs | 100 / 100 | JavaScript | 4,069 | |
🔮 A Node.js scraper for humans. What the test found: The scrape-it library installs and all 9 integration tests pass on Node.js v18.19.1 after pinning cheerio to 1.0.0. 2 minutes. | ||||||
| 16 | toapi | Runs | 100 / 100 | Python | 3,557 | |
Every web site provides APIs. What the test found: toapi 2.2.4 installed in editable mode, all 7 pytest tests pass, CLI reports version 2.2.4, ruff linting passes, and the API serves JSON responses via WebTest against example.com. 2 minutes. | ||||||
| 17 | php-curl-class | Runs | 100 / 100 | PHP | 3,293 | |
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs What the test found: PHP Curl Class library installed via composer, PHP 8.3.12 with curl 8.10.1, vendor autoload and all 66 dependencies resolved, library classes load correctly (Curl: 152 methods, MultiCurl: 74 methods), core HTTP methods and parallel requests verified working against local test server. 74 minutes. | ||||||
| 18 | google-play-scraper | Runs | 100 / 100 | JavaScript | 3,005 | |
Node.js scraper to get data from Google Play What the test found: The google-play-scraper npm module installs, its index.js exports all 12 documented methods, and all 84 tests pass against live Google Play endpoints. 3 minutes. | ||||||
| 19 | crawler | Runs | 100 / 100 | PHP | 2,827 | |
https://spatie.be/docs/crawler What the test found: 229 tests pass (552 assertions) for the Spatie Crawler package using a downloaded static PHP 8.4.23 binary. 5 minutes. | ||||||
| 20 | geziyor | Runs | 100 / 100 | Go | 2,781 | |
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering. What the test found: Every test in the repository passes: go build ./... succeeds silently; go test -count=1 ./... passes all 6 test packages (geziyor, cache, cache/diskcache, cache/leveldbcache, client, export, middleware) with no failures. 5 minutes. | ||||||
| 21 | gecco | Runs | 100 / 100 | Java | 2,510 | |
Easy to use lightweight web crawler(易用的轻量化网络爬虫) What the test found: Gecco 1.3.21 compiles cleanly, packages as a 146K jar, and the framework correctly matches URLs, converts types, builds HTTP requests, registers DynamicGecco beans, and runs a full crawl-pipeline-render cycle against a local HTTP server. 4 minutes. | ||||||
| 22 | Crawler-Detect | Runs | 100 / 100 | PHP | 2,410 | |
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent What the test found: PHP 8.3.6 with Phar/iconv/mbstring/readline/xml/dom/curl/tokenizer/xmlwriter extensions is installed via extracted debs, Composer 2.11.2 is installed, project dependencies are resolved, and the test suite passes (28 tests, 170886 assertions). 18 minutes. | ||||||
| 23 | goclone | Runs | 100 / 100 | Go | 2,304 | |
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds. What the test found: The goclone website mirroring tool builds, passes all 18 unit tests, and successfully clones real websites (verified with https://example.com). 3 minutes. | ||||||
| 24 | x-crawl | Runs | 100 / 100 | TypeScript | 1,888 | |
Flexible Node.js crawler library What the test found: x-crawl builds successfully and 26 automation tests pass covering crawlData, crawlHTML, crawlPage, async/sync mode, device fingerprint, proxy rotation, and error collection. 9 minutes. | ||||||
| 25 | ruia | Runs | 100 / 100 | Python | 1,738 | |
Async Python 3.6+ web scraping micro-framework based on asyncio What the test found: Ruia 0.8.5 installed and fully functional inside a venv at /tmp/ruia-venv; all 67 tests pass, package importable as ruia.__version__ == '0.8.5'. 12 minutes. | ||||||
| 26 | selectolax | Runs | 100 / 100 | Python | 1,717 | |
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python. What the test found: selectolax 0.4.13 builds from source with Cython and Lexbor, and all 279 tests pass. 7 minutes. | ||||||
| 27 | siteone-crawler | Runs | 100 / 100 | Rust | 1,020 | |
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization, ideal for developers, DevOps... What the test found: siteone-crawler v2.7.0-dev builds from source, passes all 130 unit tests, and crawls a local page producing valid JSON output. 4 minutes. | ||||||
| 28 | stormcrawler | Runs | 100 / 100 | Java | 1,014 | |
A scalable, mature and versatile web crawler based on Apache Storm What the test found: Apache StormCrawler 4.0.0-SNAPSHOT builds and all 18 modules compile successfully; 983/983 pass/fail/error (0/0) with 46 Docker-dependent tests skipped; core JAR at core/target/stormcrawler-core-4.0.0-SNAPSHOT.jar (404 KB). 20 minutes. | ||||||
| 29 | x-reader | Runs | 100 / 100 | Python | 992 | |
Universal content reader MCP Server for 10+ platforms What the test found: x-reader installs cleanly in a virtual environment, all 10 unit tests pass, and the CLI fetches real content from URLs via the Jina reader fallback. 2 minutes. | ||||||
| 30 | TikHub-API-Python-SDK | Runs | 100 / 100 | Python | 945 | |
High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱)... What the test found: SDK installs, imports, all 110 tests pass, and both sync/async clients successfully reach the real TikHub API health endpoint and the CLI prints version info. 2 minutes. | ||||||
| 31 | icrawler | Runs | 100 / 100 | Python | 937 | |
A multi-thread crawler framework with many builtin image crawlers provided. What the test found: icrawler 0.1.dev1 installs in Python 3.12 venv, passes 5 test cases (4 real crawler smoke tests + 1 doctest), Bing and Baidu crawlers download valid JPEG/PNG images from the web after fixing 3 runtime bugs. 12 minutes. | ||||||
| 32 | scrapyrt | Runs | 100 / 100 | Python | 884 | |
HTTP API for Scrapy spiders What the test found: ScrapyRT 0.18.1 installed in a venv, all 126 unit and integration tests pass, the HTTP server starts and responds to GET and POST crawl.json endpoints with real Scrapy spider output. 9 minutes. | ||||||
| 33 | scrapecraft | Runs | 100 / 100 | Python | 731 | |
🤖 AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph. What the test found: ScrapeCraft backend serves all API endpoints on uvicorn, WebSocket connections establish and respond, frontend builds successfully, chat uses real OpenRouter API key, JWT auth works, pipeline CRUD works, scraping endpoints respond. 14 minutes. | ||||||
| 34 | oc | Runs | 100 / 100 | JavaScript | 694 | |
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands. What the test found: only-cli v0.5.8 installs and runs on Node 20.11.1; 263 of 264 tests pass; the CLI fetches, distills, and renders public web pages with token-budgeted output and actionable numbered links. 3 minutes. | ||||||
| 35 | NetDiscovery | Runs | 100 / 100 | Java | 643 | |
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。 What the test found: The full multi-module Gradle build passes (BUILD SUCCESSFUL, 63 tasks), and the NetDiscovery spider engine example ran as a Java process answering HTTP 200 with JSON on port 8715 routes /netdiscovery/health/, /netdiscovery/spiders/, /netdiscovery/spider/tony1/detail, and /netdiscovery/metrics/, while the downloader... 69 minutes. | ||||||
| 36 | huashu-chrome | Runs | 100 / 100 | JavaScript | 610 | |
让任何 AI agent 操控你自己的 Chrome, 带着全部登录态。MCP + Chrome 扩展,Claude Code / Codex / Cursor / Gemini CLI 通用。 What the test found: All 123 unit tests pass, npm install completes, MCP server responds with 22 tools, bridge daemon starts and answers probes, doctor command runs successfully. 2 minutes. | ||||||
| 37 | freeproxy | Runs | 100 / 100 | Python | 595 | |
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,每日更新海量免费代理。) What the test found: pyfreeproxy v0.5.2 installed in /tmp/venv, all 48 proxy sources registered, Proxifly/ProxyScrape/Geonode scrape real proxies successfully, filter rules and ProxiedSessionClient work, JSON export round-trips correctly, free proxies collected but actual proxy requests time out as expected from this container network. 3 minutes. | ||||||
| 38 | social-media-profile-scrapers | Runs | 100 / 100 | Python | 588 | |
Fetch user's data across social media What the test found: 8/8 tests pass; GitHub and Pinterest CLI scrapers fetch real profile data; the 6 remaining scrapers return None gracefully given no browser binary or API access. 16 minutes. | ||||||
| 39 | reader | Runs | 100 / 100 | TypeScript | 579 | |
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents. What the test found: Reader CLI builds, all 346 unit tests pass, the daemon starts with Playwright's bundled Chromium, and an end-to-end scrape of https://example.com via HTTP returns clean HTML and markdown content with a 200 status. 13 minutes. | ||||||
| 40 | Stealth-Requests | Runs | 100 / 100 | Python | 570 | |
Undetected web-scraping & seamless HTML parsing in Python! What the test found: stealth-requests 2.1.0 installed in a fresh venv, all 64 tests pass, and real requests to httpbin.org succeed with proper Chrome impersonation headers. 1 minute. | ||||||
| 41 | ticket-monitoring | Runs | 100 / 100 | Python | 528 | |
✈️ 自托管的多渠道机票价格监控:航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线(五洲目的地)同一套采集-判定-推送链路;数据留在本机,配置全在网页,开源免费。 What the test found: All dependencies installed in venv, Playwright chromium installed, test suite runs 3712 tests all passing, web UI demo server starts and responds with HTTP 200 on its state API endpoint, main.py --version reports 1.5.303. 44 minutes. | ||||||
| 42 | fundus | Runs | 100 / 100 | Python | 480 | |
A very simple news crawler with a funny name What the test found: Fundus 0.5.7 installs cleanly in a venv, all 1131 unit tests pass, and the crawler successfully fetches live article data from The New Yorker via HTTP. 3 minutes. | ||||||
| 43 | camofox-browser | Runs | 100 / 100 | JavaScript | 453 | |
Anti-detection browser server for AI agents, REST API wrapping Camoufox engine with OpenClaw plugin support What the test found: Camofox Browser Server builds, starts on port 9998, creates browser contexts via Camoufox Firefox headless, navigates to https://example.com, returns accessibility snapshots, and passes the 677-test full suite with Node.js 20 and rebuilt native addons. 34 minutes. | ||||||
| 44 | sparkler | Runs | 100 / 100 | Python | 419 | |
A crawl workstation: View, Control, and Crawl. Solr CrawlDB, Tika, Vue 3. What the test found: Sparkler is installed with Python 3.12, Solr 10.0.0 on port 8983, Vue UI built and served on port 8180, APIs responding, all 32 unit tests passing. 7 minutes. | ||||||
| 45 | Hydra | Runs | 100 / 100 | Java | 406 | |
为超级个体和一个人公司打造一个人的大厂,Hydra九头龙构筑大规模AI调度、数据采集、情报系统、数据平台、分析决策、产品生产的'军事'工业基座。 What the test found: After fixing module-name casing, pinning Lombok 1.18.34, and adding the missing System.Subsystem config block, the Maven build succeeds and the ShadowBoot crawler app boots and runs the README's Void minimum-system demo to completion (exit 0, 'Pinecone Runtime : 136 /ms !'), while the full test suite passes with 48... 9 minutes. | ||||||
| 46 | trends-checker | Runs | 100 / 100 | Python | 405 | |
Python CLI for Google Trends analysis with advanced rate limiting, cookie auth, and beautiful terminal output. Built on pytrends with enterprise features. What the test found: trends-checker installs and runs, hitting the real Google Trends API via pytrends (with urllib3 compat patch applied to pytrends request.py), returning real interest data verified at score 42.83 for 'AI agents' in US. 5 minutes. | ||||||
| 47 | ir-search | Runs | 100 / 100 | Python | 394 | |
🇰🇷 한국 정부 지원사업 전수조사 에이전트 스킬 [Claude Code·Codex·agy(Antigravity)·Cursor·Gemini CLI·Grok Build 지원] K-Startup·기업마당·NIPA·KOCCA·SMTECH 크롤링 + 즉시/로드맵/변형 3분류... What the test found: The project's 229 pytest tests pass, and every CLI crawler (K-Startup, bizinfo, NIPA, KOCCA, SMTECH list and detail, scope_plan, diff_surveys) ran successfully against live Korean government sites, producing real announcement records and detail text. 4 minutes. | ||||||
| 48 | crawler | Runs | 100 / 100 | PHP | 373 | |
Library for Rapid (Web) Crawler and Scraper Development What the test found: 1278 of 1278 default unit tests pass, PHP 8.3.32 and Composer installed, Chrome for Testing available, integration server fixed with short_open_tag=Off and 500ms startup delay. 27 minutes. | ||||||
| 49 | scrapy-zyte-smartproxy | Runs | 100 / 100 | Python | 365 | |
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy What the test found: scrapy-zyte-smartproxy v2.4.1 installed in a Python 3.12 venv; all 82 existing unit tests pass with zero failures. 1 minute. | ||||||
| 50 | hQuery.php | Runs | 100 / 100 | PHP | 358 | |
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies. What the test found: hQuery.php library installs, autoloads, and parses HTML via fromHTML() and find()/text() selectors; all 27 unit tests pass on PHP 8.4.1 via FrankenPHP static binary. 7 minutes. | ||||||
| 51 | geo-aeo-tracker | Runs | 100 / 100 | TypeScript | 334 | |
Open-source AI visibility tracker and AEO checker. Track how ChatGPT, Perplexity, Gemini and Google AI Mode mention and cite your brand. MCP server + CLI... What the test found: Next.js dev server runs on port 3000 serving the GEO/AEO Tracker dashboard in demo mode, with all pages (/ and /demo) and API routes (/api/analyze, /api/audit, /api/scrape, etc.) responding correctly. 4 minutes. | ||||||
| 52 | aliexpress-product-scraper | Runs | 100 / 100 | JavaScript | 331 | |
Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc., What the test found: Node 24.0.0 installed, pnpm installed, Chromium 145 downloaded, all 22 tests pass, ESLint clean. 1 minute. | ||||||
| 53 | Laravel-Crawler-Detect | Runs | 100 / 100 | PHP | 326 | |
A Laravel wrapper for CrawlerDetect - the web crawler detection library What the test found: All 9 package tests pass, detecting crawlers from real user agent lists via the Laravel facade and Request macros. 4 minutes. | ||||||
| 54 | SpotifyScraper | Runs | 100 / 100 | Python | 324 | |
Extract public Spotify data, tracks, albums, artists, playlists, podcasts & lyrics, without the official API. Sync + async, typed models, one dependency. What the test found: The SpotifyScraper package is installed and working: the CLI fetches real Spotify track/artist/playlist data over the network, the hermetic unit test suite passes completely (921/921), and the MCP server imports and launches correctly after pinning mcp to v1. 3 minutes. | ||||||
| 55 | 4chan-downloader | Runs | 100 / 100 | Python | 312 | |
Python3 script to continuously download all images/videos of multiple 4chan threads simultaneously - without installation What the test found: Both scripts compile, install cleanly via venv, and successfully interact with live 4chan API endpoints: thread-watcher queries the catalog and writes URLs, inb4404 downloads thread media files with validated content. 4 minutes. | ||||||
| 56 | xvideos | Runs | 100 / 100 | TypeScript | 310 | |
xvideos API library What the test found: npm install completes with warnings. Build succeeds. 5 unit test files (47 tests) pass on fixture HTML. Full test suite: 52 pass, 4 fail (all integration tests blocked by xvideos.com returning 404 on /verified/videos/). 5 minutes. | ||||||
| 57 | facebook-ads-library-mcp | Runs | 100 / 100 | Python | 304 | |
Free, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research, advertiser, copy, landing pages, CTAs, any... What the test found: MCP server loads, imports, starts over stdio, registers both tools, and all 8 offline parser tests pass. 7 minutes. | ||||||
| 58 | teracrawl | Runs | 100 / 100 | TypeScript | 291 | |
High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative What the test found: Teracrawl server starts on port 8085 with a local Chromium-backed mock Browser.cash pool and a mock SERP service. GET /health returns 200, POST /scrape converts a live URL to markdown via real headless Chromium, POST /serp/search returns mock search results, and POST /crawl combines both to scrape multiple URLs. 15 minutes. | ||||||
| 59 | OddsHarvester | Runs | 100 / 100 | Python | 276 | |
A python app designed to scrape and process sports betting data directly from oddsportal.com 🎯 What the test found: OddsHarvester v0.15.0 installed from source with uv and Playwright; all 1308 unit tests pass and 63 of 82 integration tests pass against HAR replay (19 live-only skipped, 1 xfail for known in-play replay limitation). 40 minutes. | ||||||
| 60 | go-movies | Runs | 100 / 100 | Go | 275 | |
golang spider Crawler 爬虫 电影 What the test found: GoMovies binary compiled and serves HTTP on port 8899 with Redis persistence; spider successfully fetches movie data from hhzyapi.com external API. 20 minutes. | ||||||
| 61 | Project-Eyes-On | Runs | 100 / 100 | Python | 256 | |
Project Eyes On is a high-speed, multi-threaded surveillance tool by Y0oshi (@rde0) for locating open IP cameras worldwide. Unifies Google Dorking and... What the test found: Project Eyes On installed, imports correctly, Insecam scraping and multi-engine dorking both find real live IP cameras, and JSON/CSV/HTML reports all produce valid output. 2 minutes. | ||||||
| 62 | Uni-CLI | Runs | 100 / 100 | TypeScript | 229 | |
One command for every interface, search, run, and inspect real software across APIs, browsers, desktops, local tools, and MCP. What the test found: Uni-CLI v1.2.1 builds from source, runs 3735 passing unit tests, and executes live commands (hackernews top, search, doctor) against real remote APIs on Node 22.23.4 (npm v10.9.9). The npm global package v1.3.0 is also installed and functional. 20 minutes. | ||||||
| 63 | patchright-python | Runs | 98 / 100 | Python | 1,603 | |
Undetected Python version of the Playwright testing and automation library. What the test found: patchright installed from PyPI, builds from source, and launches patched Chromium headless shell; the package passes 584 of 590 Playwright chromium sync tests with only SSL-cert infrastructure and Xvfb-signal-handling failures. 10 minutes. | ||||||
| 64 | Douyin_TikTok_Download_API | Runs | 97.3 / 100 | Python | 21,518 | |
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video... What the test found: All 2960 tests pass (2450 unit + 510 integration), the CLI outputs version 5.0.1 and lists 10 commands, the API server starts and serves healthz/readyz/swagger/console root with 200, the web console SPA is built, and both the PostgreSQL+TimescaleDB and Redis backends and the Go downloader sidecar run. 48 minutes. | ||||||
| 65 | camofox-browser | Runs | 96.7 / 100 | JavaScript | 12,456 | |
Stealth headless browser for AI agents, bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement. What the test found: Camofox browser server installed and verified: Node 22.23.4 in user space, Camoufox 1.3GB binary downloaded, GeoIP 50MB database downloaded, server starts at localhost:9377 with Xvfb virtual display, creates browser tabs navigating to real URLs, returns accessibility snapshots with element refs, and closes cleanly on... 13 minutes. | ||||||
| 66 | browserless | Runs | 96.7 / 100 | JavaScript | 1,846 | |
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API. What the test found: All 10 of 14 testable packages pass fully (330 tests) on Node v24.0.0 with Puppeteer 25.17.0 and Chromium 157.0.8103.50; the sole regression is the navigator.languages evasion returning an extra language tag due to the newer Chrome version. 26 minutes. | ||||||
| 67 | Scrapling | Runs | 96 / 100 | Python | 104,979 | |
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ... What the test found: Scrapling 0.4.15 is installed in a venv with all extras, Playwright Chromium browser available, all 985 tests pass, CLI and MCP server commands respond correctly, and HTTP fetches via StealthyFetcher return status 200 from real endpoints. 8 minutes. | ||||||
| 68 | proxy_pool | Runs | 96 / 100 | Python | 23,800 | |
Python ProxyPool for web spider What the test found: The ProxyPool Flask API server runs on port 5010 backed by real Redis, responds to all endpoints (/get, /pop, /all, /delete, /count), and all 248 automated tests pass. 6 minutes. | ||||||
| 69 | Noria | Runs | 96 / 100 | JavaScript | 4,623 | |
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices... What the test found: All 232 tests pass across 17 suites; the Node.js pipeline orchestrator boots, loads pipelines, and displays the WhatsApp QR code; the Streamlit Control Center starts on :8501; ESLint passes; Prettier reports 2 non-critical formatting warnings in src/infrastructure/messaging/channel-fetcher.js and src/launcher.js. 3 minutes. | ||||||
| 70 | reverse-api-engineer | Runs | 96 / 100 | Python | 4,175 | |
Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor and OpenCode. What the test found: All dependencies install cleanly via pip, the CLI loads and responds to --version/--help/list--json, and the full pytest suite (1031 tests) passes. 7 minutes. | ||||||
| 71 | owllook | Runs | 96 / 100 | Python | 2,900 | |
owllook-小说搜索引擎 What the test found: The owllook web server launches on port 8001 and responds with 200 HTTP status on /, /novels/search, /novels/top, /api/search, and /health routes, with real HTML content, connecting to running Redis and MongoDB instances. The existing pytest test suite passes (1 passed, 0 failed). 15 minutes. | ||||||
| 72 | diskover-community | Runs | 96 / 100 | PHP | 1,831 | |
Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch What the test found: Elasticsearch 7.13.1 cluster green on localhost:9200, Python 3.12 virtualenv with diskover scanner indexes files to ES, and PHP 8.2.32 built-in server serves the diskover-web UI authenticating with default credentials diskover/darkdata showing both indexed indices. 23 minutes. | ||||||
| 73 | finvizfinance | Runs | 96 / 100 | Python | 1,790 | |
Finviz analysis python library. What the test found: finvizfinance v1.5.0 installed and builds. Quote, calendar, insider, news, group overview, crypto, forex, and futures all verified working against live finviz.com. Screener and earnings remain broken due to finviz SPA migration requiring a larger rewrite. 13 minutes. | ||||||
| 74 | openserp | Runs | 96 / 100 | Go | 1,608 | |
Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page... What the test found: OpenSERP builds from source, all 12 unit test suites pass, the HTTP server serves /health (200) and handles live DuckDuckGo searches (200) returning structured JSON with organic results and SERP features. 6 minutes. | ||||||
| 75 | WebAI2API | Runs | 96 / 100 | JavaScript | 1,428 | |
WebAI2API: 基于 Camoufox 的网页 AI 转 API 工具,支持 LMArena/Gemini等,多窗口并发与账号隔离。 | Web AI to OpenAI API via Camoufox. Supports LMArena/Gemini and more, multi-window... What the test found: The server starts, launches a Camoufox browser instance in headful mode on Xvfb, serves the OpenAI-compatible API on port 3000 returning 200 for /v1/models and WebUI, the SQLite history database initializes, all 19 adapters load, and the browser fingerprint is generated and saved. 6 minutes. | ||||||
| 76 | httpcloak | Runs | 96 / 100 | Go | 1,388 | |
Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic... What the test found: The httpcloak Go library builds and all 8 test packages pass cleanly. Example programs compile without vet warnings. 7 minutes. | ||||||
| 77 | wombat | Runs | 96 / 100 | Ruby | 1,360 | |
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. What the test found: Wombat 3.3.0 builds and installs as a gem under Ruby 3.2.3, all 56 RSpec tests pass, and a live Wombat.crawl scrape of a local WEBrick page returns correctly parsed structured data. 12 minutes. | ||||||
| 78 | newspaper4k | Runs | 96 / 100 | Python | 1,170 | |
📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites. What the test found: newspaper4k 0.9.6 is pip-installed and editable in /work/repo/venv; CLI runs (help + real article extraction into JSON), 234 tests pass with 2 deselected stale summarization fixtures and 5 external-site failures; package state is clean working tree. 38 minutes. | ||||||
| 79 | parse-video | Runs | 96 / 100 | Go | 1,104 | |
Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频... What the test found: The project builds, all tests pass (0 cmd tests, 0 parser tests with zero failures), the serve subcommand starts and responds on HTTP, the version subcommand prints 'parse-video dev', and the parse API returns correct error responses. 3 minutes. | ||||||
| 80 | seonaut | Runs | 96 / 100 | Go | 873 | |
Open source SEO audit tool. What the test found: SEOnaut installed from source (Go build OK, frontend CSS built, migrations applied to a local MariaDB) and ran on port 9000, serving sign-in (200), creating a user and project, completing a real crawl of 2 URLs, and rendering the dashboard. 9 minutes. | ||||||
| 81 | primp | Runs | 96 / 100 | Rust | 635 | |
HTTP client that can impersonate web browsers What the test found: The repo builds and runs: Rust workspace checks and per-crate test suites pass (1354 tests), the primp Python extension installs into crates/primp-python/.venv, 493 Python tests pass, and a real impersonated HTTPS request returns HTTP 200. 15 minutes. | ||||||
| 82 | xxl-crawler | Runs | 95.7 / 100 | Java | 756 | |
A lightweight web crawler framework.(Java爬虫框架) What the test found: XXL-CRAWLER builds and packages successfully; all 9 runnable tests pass including 4 real internet crawls against cnblogs.com, baidu.com, and 163.com with live data returned. 8 minutes. | ||||||
| 83 | patchright-nodejs | Runs | 95 / 100 | TypeScript | 813 | |
Undetected NodeJS version of the Playwright testing and automation library. What the test found: npm install and tsc --noEmit pass; patchright_rebranding.ts transforms Playwright 1.65.1 into Patchright npm packages; the patched CLI returns version 1.65.1 and chromium browser can be installed and launched headlessly via the patched API, navigating to pages and evaluating JavaScript correctly. 9 minutes. | ||||||
| 84 | pinchtab | Runs | 94.3 / 100 | Go | 10,603 | |
High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard. What the test found: A source-built PinchTab server (binary reports pinchtab dev) is running on 127.0.0.1:9867 with a responsive always-on headless instance and answered authenticated health, navigate, snapshot, text, title, and screenshot API calls with HTTP 200, while the full Go unit suite (9166 passed, 18 skipped) and scripts/check.sh... 19 minutes. | ||||||
| 85 | Beanbun | Runs | 93.5 / 100 | PHP | 1,258 | |
Beanbun 是用 PHP 编写的多进程网络爬虫框架,具有良好的开放性、高可扩展性,基于 Workerman。 What the test found: Beanbun 1.0.4 with vendored workerman and guzzle dependencies. PHP 8.3.6 extracted from apt. All source and vendor files parse without errors. Framework class instantiation, configuration, and all Helper methods (randUserAgent, formatUrl, getUrlByHtml) verified working. 16 minutes. | ||||||
| 86 | deepcrawl | Runs | 93.3 / 100 | TypeScript | 704 | |
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself. What the test found: Dependencies installed, SDK tests pass (41/41), full monorepo build and typecheck pass, and the dashboard serves HTTP 200 on / and /docs in both dev and production modes; the V0 API worker served 200 on /, /docs and /openapi in local mode with D1 migrations applied. 33 minutes. | ||||||
| 87 | rod | Runs | 91.3 / 100 | Go | 7,161 | |
A Chrome DevTools Protocol driver for web automation and scraping. What the test found: Rod project builds, rod-manager binary runs, and full test suite executes against real Chromium 128.0.6568.0 with 211/217 passing. 65 minutes. | ||||||
| 88 | nanobrowser | Runs | 90.7 / 100 | TypeScript | 14,368 | |
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks. What the test found: Dependencies installed, project builds to dist/ (background.iife.js 1,848.81 kB, manifest.json), TypeScript type-check passes across all 15 packages, unit tests pass (14/14). 15 minutes. | ||||||
| 89 | firecrawl-mcp-server | Runs | 90.2 / 100 | JavaScript | 7,884 | |
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any... What the test found: Firecrawl MCP Server v3.24.1 builds, installs, and starts successfully with 25 tools in stdio mode; 81 of 84 tests pass. 24 minutes. | ||||||
| 90 | scrapy | Runs | 90 / 100 | Python | 65,539 | |
Scrapy, a fast high-level web crawling & scraping framework for Python. What the test found: Scrapy 2.19.0 installed from source, 4539 tests pass, scrapy bench crawls 1781 pages against a local HTTP server. 15 minutes. | ||||||
| 91 | lux | Runs | 90 / 100 | Go | 31,855 | |
👾 Fast and simple video download library and CLI tool written in Go What the test found: Lux builds and runs: the binary outputs version info and help, extracts real YouTube video metadata, and 14 of 48 test packages pass (all failures are network-dependent extractor tests or DNS-timeout content-type lookups; no code defects remain). 11 minutes. | ||||||
| 92 | news-please | Runs | 90 / 100 | Python | 2,491 | |
news-please - an integrated web crawler and information extractor for news that just works What the test found: news-please 1.6.13 installs and its library API extracts structured article data (title, authors, publication date, description, maintext, language) from HTML via both HTTP and raw HTML input; CLI entry points respond with help output. 5 minutes. | ||||||
| 93 | top-user-agents | Runs | 90 / 100 | JavaScript | 385 | |
Always up-to-date list of the top 100 most common browser user-agents for HTTP clients. What the test found: The repo installs cleanly; with the bundled prebuilt Node 22, npm test passes all 5 tests and require/consumer checks return 100 main, 82 desktop, and 18 mobile valid user-agents from the packed tarball; only the Redis-backed update scripts remain unrunnable in this container because no redis-server is available. 3 minutes. | ||||||
| 94 | skills | Runs | 85 / 100 | Python | 6,417 | |
Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution... What the test found: browser-act-cli 1.4.2 installed, skill handshake completed, and the chrome browser type (bundled Chromium) works end-to-end, browser create, open, navigate, state inspection, content extraction via get markdown, and session close all succeed; stealth features require a server-side-validated BrowserAct API key that... 5 minutes. | ||||||
| 95 | nebula | Runs | 82.5 / 100 | Go | 380 | |
🌌 An agnostic network crawler exposing comprehensive peer information and network topology information. What the test found: Nebula builds with just build, runs dry-run and ClickHouse-backed IPFS crawls against real network peers, persists crawl data to ClickHouse, and passes all 92 tests across 7 packages with Postgres 16.4 and ClickHouse 26.11 running locally on their test ports. 16 minutes. | ||||||
| 96 | browser-use | Runs | 66.7 / 100 | Python | 121,752 | |
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API. What the test found: browser-use v0.13.10 installed in a Python 3.12 venv with uv, Chromium headless installed via playwright, and all 563 CI tests pass (529 passed, 34 skipped) after fixing 2 test identity comparison bugs in test_beta_agent.py that only manifest under xdist worker parallelism. 75 minutes. | ||||||
| 97 | SurfSense | Runs | 66.3 / 100 | Python | 16,927 | |
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord... What the test found: All Python trees (surfsense_backend, surfsense_mcp, surfsense_local/backend, plugins/remote/host core tests) install and pass tests; the web frontend fails to compile native deps on Node 18.x (requires 24.x); docs are clean. 20 minutes. | ||||||
| 98 | changedetection.io | Runs | 60 / 100 | Python | 35,963 | |
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock... What the test found: changedetection.io 0.60.7 installs into a venv, launches and serves HTTP 200, its full basic test suite passes (680 passed in the main group, exit 0, plus 508 unit tests), and a real watch added via the API detected a content change and exposed it in the UI. 76 minutes. | ||||||
| 99 | jsoup | Runs | 60 / 100 | Java | 11,418 | |
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety. What the test found: jsoup compiles from source with JDK 21 and Apache Maven, all 2,341 unit tests pass, and the built JAR correctly parses HTML and extracts DOM content. 2 minutes. | ||||||
| 100 | JMComic-Crawler-Python | Runs | 60 / 100 | Python | 8,165 | |
Python API for JMComic | 提供Python API访问禁漫天堂,同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀 What the test found: Package jmcomic 2.7.7 installed in a Python 3.12.3 venv with all dependencies; 238/240 tests pass including real network calls against JMComic API servers; CLI commands jmcomic and jmv produce correct help output. 10 minutes. | ||||||
| 101 | stealth-browser-mcp | Runs | 60 / 100 | Python | 3,385 | |
Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning. What the test found: Stealth Browser MCP server runs on stdio transport, spawns a headless Chromium 157.0.8086.0, navigates to data URLs, extracts page HTML content, and closes cleanly -- all via MCP protocol over stdio. 15 minutes. | ||||||
| 102 | spider | Runs | 60 / 100 | Rust | 2,857 | |
Foundational low latency web data collecting in Rust What the test found: Rust 1.99.0 installed, cargo build succeeds, spider CLI binary crawls and scrapes example.com returning status 200 and expected page content, all 608 spider core tests pass, workspace-wide crate tests all pass. 56 minutes. | ||||||
| 103 | google-maps-scraper-kit | Runs | 60 / 100 | Python | 2,581 | |
Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and... What the test found: The scraper binary runs natively, its REST API on localhost:8080 creates/polls/downloads scraping jobs against real Google Maps data, and the included Python CLI successfully produces lead CSVs with real business listings. 7 minutes. | ||||||
| 104 | gain | Runs | 60 / 100 | Python | 2,017 | |
Web crawling framework based on asyncio. What the test found: All 7 tests pass, ruff lint passes, library imports correctly, and item attribute extraction from HTML produces expected results. 1 minute. | ||||||
| 105 | wreq-python | Runs | 60 / 100 | Python | 1,529 | |
An ergonomic, privacy-aware Python HTTP Client What the test found: wreq Python package builds, installs, and passes all 57 unit tests against a local httpbin server, supporting HTTP/1.1 and HTTP/2 requests, cookie jars, redirect policies, TLS emulation (JA3/JA4), multipart uploads, streaming, and compression. 15 minutes. | ||||||
| 106 | rebrowser-patches | Runs | 60 / 100 | JavaScript | 1,471 | |
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to... What the test found: rebrowser-patches repo installs, builds 6 drop-in tarballs, and passes all 43 test gates (detection, scenarios, smoke) with patched Puppeteer-core 25.14.0 and Playwright-core 1.64.1 defeating Runtime.Enable detection. 8 minutes. | ||||||
| 107 | fscrawler | Runs | 60 / 100 | Java | 1,459 | |
Elasticsearch File System Crawler (FS Crawler) What the test found: FSCrawler 3.1-SNAPSHOT builds successfully via Maven, distributable ZIP is packaged, binary launches and shows help/version, and 687 unit tests pass when run with the `-Dtest="*Test"` workaround. 82 minutes. | ||||||
| 108 | crawly | Runs | 60 / 100 | Elixir | 1,129 | |
Crawly, a high-level web crawling & scraping framework for Elixir. What the test found: Crawly 0.17.2 Elixir library compiles, starts its Cowboy HTTP API on port 4001 answering 200, and runs its 136-test suite with 0 failures and 1 excluded test. 11 minutes. | ||||||
| 109 | wreq | Runs | 60 / 100 | Rust | 1,121 | |
An ergonomic, privacy-aware Rust HTTP Client What the test found: wreq v0.16.1 builds and all 280+ integration tests pass on Rust 1.99.0 (Ubuntu 24.04) with BoringSSL via btls, after providing libclang (extracted from libclang1-20 + libclang-20-dev debs) and workaround for Ubuntu 24.04's missing /usr/include/stddef.h. 35 minutes. | ||||||
| 110 | jvppeteer | Runs | 60 / 100 | Java | 805 | |
Java API For Chrome and Firefox What the test found: Jvppeteer 3.6.5 builds and installs via Maven, Chrome for Testing 149.0.7798.0 runs headlessly, and the library launches the browser, navigates pages, evaluates JavaScript, and retrieves page content successfully. 26 minutes. | ||||||
| 111 | agent-fetch | Runs | 60 / 100 | TypeScript | 316 | |
Full-content web fetcher for AI agents, Chrome TLS fingerprinting, browser impersonation, multi-strategy article extraction, and web scraping. What the test found: agent-fetch 0.1.6 builds from source, passes all 904 unit tests, and successfully fetches and extracts real web pages (Wikipedia, example.com) via CLI. 6 minutes. | ||||||
| 112 | crawl4ai | Runs | 45.7 / 100 | Python | 87,664 | |
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this... What the test found: Crawl4AI 0.9.4 installed in venv, Playwright chromium browser downloaded, DB initialized, AsyncWebCrawler can crawl real URLs with markdown output, cache operations work, and 360 core tests pass with 11 skipped. 14 minutes. | ||||||
| 113 | node-crawler | Runs with mocks | 92 / 100 | TypeScript | 6,797 | |
Web Crawler/Spider for NodeJS + server-side jQuery ;-) What the test found: installed deps with local Node v22.14.0, built dist, and the full test suite passes (51/51) with HTTP/2 tests served by a local TLS HTTP/2 mock after nghttp2.org returned 502. 54 minutes. | ||||||
| 114 | twscrape | Runs with mocks | 92 / 100 | Python | 2,985 | |
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling. What the test found: twscrape 0.20.1 is installed in a Python venv, its CLI answers the version command (twscrape: 0.20.1, SQLite runtime 3.45.1), and its mock-backed pytest suite passes all 217 tests. 2 minutes. | ||||||
| 115 | brightdata-mcp | Runs with mocks | 92 / 100 | JavaScript | 2,733 | |
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access. What the test found: All 21 unit tests pass with Node 20.20.2. The MCP server loads fastmcp without errors and rejects missing API_TOKEN. Dependencies installed cleanly. 2 minutes. | ||||||
| 116 | boss-zhipin-scraper | Runs with mocks | 92 / 100 | Python | 1,784 | |
Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler. What the test found: All 109 mocked tests pass, both scripts compile, CLI help and city listing work, version is consistent across all 5 files, and the environment check correctly reports the absence of a real Chrome CDP endpoint. 1 minute. | ||||||
| 117 | Scweet | Runs with mocks | 92 / 100 | Python | 1,667 | |
Twitter scraper and X scraper for Python: scrape tweets, profiles, followers, following and trends without the official API. Twitter scraping library with... What the test found: Scweet 5.8.0 installed from source, all 495 unit tests pass, CLI help works, package imports correctly. The library requires real X accounts and auth tokens for live scraping. 12 minutes. | ||||||
| 118 | fredy | Runs with mocks | 92 / 100 | JavaScript | 1,650 | |
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24... What the test found: Fredy backend serves UI on port 9998 with all 45 migrations applied, login authenticates admin/admin, and all 4260 offline tests pass. 5 minutes. | ||||||
| 119 | x-kit | Runs with mocks | 92 / 100 | Python | 947 | |
Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets What the test found: uv sync installs 14 packages, all 7 main scripts import without error, 5 of 7 scripts run against a local mock X.com API server returning realistic GraphQL payloads; real X.com credentials (auth_token + ct0) needed for live production use. 8 minutes. | ||||||
| 120 | chatWeb | Runs with mocks | 92 / 100 | Python | 916 | |
ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points. What the test found: ChatWeb launches in both API mode (FastAPI/uvicorn on :9531) and WebUI mode (Gradio on :7860). File upload, summary generation, and Q&A endpoints all return HTTP 200 with correct JSON schema against a mock OpenAI API. The three startup modes (console/api/webui) are all functional; console mode was not interactively... 9 minutes. | ||||||
| 121 | Scavenger | Runs with mocks | 92 / 100 | Python | 884 | |
Crawler (Bot) searching for leaks on paste sites. What the test found: The app installs cleanly, all 6 Python modules compile without errors, the core credential detection correctly identifies email:password pairs and sensitive keyword matches via configurable search terms and regex patterns, the SQLite paste-ID dedup tracker and file archiver both function correctly, and the dashboard... 5 minutes. | ||||||
| 122 | spidr | Runs with mocks | 92 / 100 | Ruby | 835 | |
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. What the test found: Spidr 0.7.2 on Ruby 3.3.6: 317 RSpec tests pass with 0 failures, gem dependencies installed, web spidering via WebMock-stubbed Sinatra apps works correctly. 33 minutes. | ||||||
| 123 | everyinfra-agent-plugins | Runs with mocks | 92 / 100 | Python | 697 | |
EveryInfra MCP server & agent plugin: scraper API for TikTok, Instagram, Amazon, YouTube, Xiaohongshu & Douyin. 612 data APIs across 106 platforms, from... What the test found: EveryInfra plugin manifests, skills, MCP configs, marketplace sources and build scripts validate cleanly; deterministic release archive and 13 host-package ZIPs build successfully; the example proxy+data API flow runs correctly against local mocks. 3 minutes. | ||||||
| 124 | catbus | Runs with mocks | 92 / 100 | TypeScript | 642 | |
一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用 What the test found: catbus CLI is installed, builds, and all 962 offline tests pass. The binary outputs valid JSON envelopes for version, help, platforms, doctor, and handles platform aliases, native name mapping, and auth-required errors correctly. 3 minutes. | ||||||
| 125 | browser-search | Runs with mocks | 92 / 100 | JavaScript | 550 | |
A skill for AI agents: search the web with SearXNG, browse with Camofox, bypass protections with CloakBrowser. Anti-hallucination by design. Self-hosted, free... What the test found: npm install succeeds, CloakBrowser (with Node 20, not the system Node 18) fetches live pages, the SearXNG and Camofox deterministic scripts pass their full command flows against schema-correct mocks, unit suites pass 28/28 and 12/12, and setup.sh/audit.sh complete without failed checks; real SearXNG/Camofox Docker... 33 minutes. | ||||||
| 126 | scrapple | Runs with mocks | 92 / 100 | Python | 506 | |
A framework for creating semi-automatic web content extractors What the test found: Scrapple 0.3.0 installs, builds, and runs on Python 3.12; the CLI accepts all 4 commands (genconfig, run, generate, web), scrapes live websites via XPath and CSS selectors, and generates executable Python scripts, 13/17 tests pass with the remaining 4 failing only because their target websites changed HTML since 2018. 10 minutes. | ||||||
| 127 | wreq-js | Runs with mocks | 92 / 100 | TypeScript | 464 | |
HTTP client for Node.js with browser TLS fingerprint impersonation What the test found: wreq-js v3.2.0 is installed and working on x86_64 Ubuntu 24.04 with Node.js v22.23.4, the TypeScript build completes, the native Rust binding (prebuilt) loads correctly, and the test suite passes 195/200 tests against a local HTTP/HTTPS/WebSocket test server. 5 tests are skipped: 4 requiring full native setCookies... 28 minutes. | ||||||
| 128 | email-sleuth | Runs with mocks | 92 / 100 | Rust | 430 | |
Discover and verify professional emails using names + domains What the test found: Project builds from source, 9 unit tests pass, and both email-sleuth/es binaries execute their full pipeline (config loading, SMTP connectivity testing, pattern generation, DNS resolution, JSON output) without crashing. 6 minutes. | ||||||
| 129 | google-news-scraper | Runs with mocks | 92 / 100 | JavaScript | 384 | |
Lightweight scraper for Google News What the test found: TypeScript compiles cleanly, rollup builds CJS+ESM bundles, both module formats load at runtime, and all 46 unit/integration tests pass on Node 18.19.1. 13 minutes. | ||||||
| 130 | news-crawl | Runs with mocks | 92 / 100 | Java | 384 | |
News crawling with StormCrawler - stores content as WARC What the test found: The NewsCrawlBootstrap topology runs in Storm LocalCluster against local mocks, detects the RSS feed, acks status updates to OpenSearch, and writes a gzip-valid WARC containing fetched feed content; mvn clean package and all 10 unit tests pass. 69 minutes. | ||||||
| 131 | graphlit-mcp-server | Runs with mocks | 92 / 100 | TypeScript | 379 | |
Model Context Protocol (MCP) Server for Graphlit Platform What the test found: Graphlit MCP Server builds, launches, and responds correctly to MCP protocol initialize, resources/list, and tools/list requests. 10 minutes. | ||||||
| 132 | twitter-scraper-selenium | Runs with mocks | 92 / 100 | Python | 344 | |
Python's package to scrap Twitter's front-end easily What the test found: Package installs in a virtualenv, all 9 tests pass, xquik search function verified end-to-end with a mock HTTP server. 2 minutes. | ||||||
| 133 | gplay-scraper | Runs with mocks | 92 / 100 | Python | 315 | |
GPlay Scraper is a powerful Python Google Play scraper library for extracting comprehensive app data from the Google Play Store. Scrape Google Play Store apps... What the test found: gplay-scraper installed in a venv with all dependencies; all 48 tests pass after fixing the tls_client client_identifier string from 'chrome112' to 'chrome_112'. 4 minutes. | ||||||
| 134 | MinerU-HTML | Runs with mocks | 92 / 100 | Python | 302 | |
MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data... What the test found: MinerU-HTML v1.1.2 package installs and imports correctly in a Python 3.12 venv; the full extraction pipeline runs to completion with a mock LLM backend, correctly filtering navigation/auxiliary elements from main content HTML; all 20 internal modules import cleanly; config validation, fallback handlers... 16 minutes. | ||||||
| 135 | laravel-seo-scanner | Runs with mocks | 92 / 100 | PHP | 275 | |
Scan your Laravel application routes for SEO improvements suggestions. What the test found: composer install works, the full Pest suite passes (171 passed, 3 skipped), PHPStan passes, and seo:scan-url returns real results (score 65, 28 passed, 15 failed) plus seo:scan persists the score to MariaDB, all against local mocks in this container. 28 minutes. | ||||||
| 136 | examples | Runs with mocks | 92 / 100 | TypeScript | 241 | |
Examples for using Hyperbrowser What the test found: After npm install, TypeScript compiles cleanly and the company-researcher CLI runs end-to-end against a schema-correct local Hyperbrowser mock, producing the full structured research output and exiting 0; real Hyperbrowser/OpenAI API keys were not exercised. 12 minutes. | ||||||
| 137 | google-maps-scraper | Runs with mocks | 86 / 100 | Go | 8,319 | |
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude... What the test found: All 17 Go packages with tests passed with race detection (no failures), the binary compiled and prints version/help correctly, and all 10 JS skill tests plus shell helper tests passed. 6 minutes. | ||||||
| 138 | awesome-agent-apis | Runs with mocks | 86 / 100 | Python | 1,070 | |
660+ muapi-hosted generative-media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping), one YAML file per entry... What the test found: PyYAML 6.0.3 installed; catalog_validate.py validates all 665 YAML entries in providers/ and models/ successfully with zero errors. 3 minutes. | ||||||
| 139 | extractor | Runs with mocks | 72 / 100 | TypeScript | 322 | |
Use LLMs to robustly extract web data What the test found: npm install builds successfully, 84 unit tests pass, 63 html-to-markdown integration tests pass with real ground-truth data, all LLM extraction integration tests pass with mocked providers, processedContent tests pass; Playwright browser tests fail at import on Node 18. 39 minutes. | ||||||
| 140 | Scrapegraph-ai | Runs with mocks | 56 / 100 | Python | 32,890 | |
Python scraper based on AI What the test found: 191 unit tests pass after fixing 13 categories of test failures covering models_tokens assertions, cleanup_html behaviors, logger capture, file paths, missing dependencies, mock patterns, and mock class completeness. 46 minutes. | ||||||
| 141 | Decodo | Runs with mocks | 56 / 100 | Java | 1,264 | |
HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information. What the test found: Mock proxy verified: shell/shell.sh (curl via proxy): returned 200 with IP JSON; python/python.py (requests via proxy): returned 200 with IP JSON; nodejs/node.js (axios via proxy): returned 200 with IP JSON. 9 minutes. | ||||||
| 142 | gpt-promo-scanner | Runs with mocks | 56 / 100 | Python | 948 | |
ChatGPT Team(Business) 促销码自动扫描工具, 批量发现/验证/价格收集,支持 17 国 34 个码,最高折扣 71% | ChatGPT Business promo code scanner, batch discovery, validation, price collection, 34... What the test found: Repo installs cleanly and every scanning script (auto_scan, discover_codes, verify, open_stripe, partner_scan, and the us/mega/uk scanners) ran and produced valid output against the local mock of the ChatGPT API and Clash proxy; open_stripe.py works after switching its hardcoded proxy to the project config-driven... 43 minutes. | ||||||
| 143 | agent-skills | Runs with mocks | 56 / 100 | JavaScript | 869 | |
Official Agent skills of Oxylabs products What the test found: Five Oxylabs agent skills (headless-browser, proxies, video-data, web-scraper-api, web-unblocker) install via skills CLI. .env.example corrected to OXY_RES_USERNAME/OXY_RES_PASSWORD consistent with all docs. Mock server confirms the documented API endpoint patterns respond 200 OK. 7 minutes. | ||||||
| 144 | google-search-results-python | Runs with mocks | 56 / 100 | Python | 762 | |
Google Search Results via SERP API pip Python Package What the test found: The SerpApi Python client library installs from source, runs 43 unit tests successfully against a locally hosted mock backend, and passes a basic smoke test (OOBT). 11 minutes. | ||||||
| 145 | google-reviews-scraper-pro | Runs with mocks | 56 / 100 | Python | 350 | |
A powerful Google Maps review scraper that works in 2026. Extracts multi-language reviews with images, handles MongoDB integration, and bypasses detection... What the test found: Python venv setup complete, all dependencies installed, 449/462 tests pass, API server runs and responds 200 on health and data endpoints, all CLI commands operational. 17 minutes. | ||||||
| 146 | quantumproxies.io | Runs with mocks | 51 / 100 | Java | 186 | |
Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries, plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#... What the test found: Python and Node.js quickstarts and curl HTTP-examples work through mock proxy to real ipinfo.io; PHP/Go/Ruby/Java/C# examples not tested due to missing runtimes; SOCKS5 not tested due to minimal mock. 5 minutes. | ||||||
| 147 | WechatSogou | Could not verify | 80 / 100 | Python | 6,411 | |
基于搜狗微信搜索的微信公众号爬虫接口 What the test found: wechatsogou 4.5.4 is installed in a Python 3.12 venv; pip install -e . succeeds; 24 offline tests pass (constants, tools, URL generation, HTML parsing); real-network API tests fail only because DNS is unavailable in the container. 13 minutes. | ||||||
| 148 | fess | Could not verify | 47.5 / 100 | Java | 1,152 | |
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG &... What the test found: Fess 15.9.0-SNAPSHOT builds and packages successfully from source. All 7,634 unit tests pass. The embedded Tomcat binds port 8080 but the application fails at runtime because no OpenSearch backend is reachable on localhost:9200. 36 minutes. | ||||||
| 149 | agentql | Could not verify | 20 / 100 | Python | 1,510 | |
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and... What the test found: could not be verified; the log shows where it stopped. | ||||||
| 150 | Krawl | Could not verify | 10 / 100 | Python | 844 | |
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities... What the test found: Krawl standalone mode builds and runs: after pip install -r requirements.txt, uvicorn app:app --app-dir src serves HTTP 200 on /dashf/healthz, answers the honeypot with 204 on /, records POST credentials from 198.51.100.77 into SQLite, and the dashboard API shows that IP categorized as attacker. 92 minutes. | ||||||
| - | moli | Not yet tested | - | Rust | 24,245 | - |
| - | scylla | Not yet tested | - | Python | 4,021 | - |
| - | ketch | Not yet tested | - | Go | 756 | - |
| - | tiktok-web-signature | Not yet tested | - | Python | 445 | - |
runs installed and started with its real dependencies. runs with mocks started after stand-ins replaced external services such as a database or a third-party API. could not verify neither the standard agent nor the stronger one got it running within the time limit; the log shows where it stopped.
How we tested
On this list as of the latest test: 112 projects ran as-is, 34 with mocks, 4 could not be verified, 4 still waiting. Languages tested: Elixir, Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript. Every attempt used a clean single-use machine, the subject at a pinned version, and a 90-minute limit for standard projects; the complete procedure is on the methodology page.
Frequently asked questions (FAQs)
How is this list ranked?
By measurement, not opinion: projects Argusic installed and launched on a fresh machine come first, then those that ran with mocks in place of external services, then those it could not verify. Ties go to the Argusic Score, then how popular it is on its own source.
Why are some projects unranked?
4 projects are still waiting for a test or for a finished attempt. They are listed without a rank until Argusic has measured them.
Where is the evidence?
Every row links to the project's Argusic page, where each run has a full log and a terminal recording stored with a sha256 fingerprint. The same pages exist for every one of the tested projects, on this list or not.
More lists in this category
- browser automation tools (shares Uni-CLI, WebAI2API, brightdata-mcp, browser-search, browser-use, camofox-browser, camofox-browser, ferret, huashu-chrome, moli, nanobrowser, oc, openserp, pinchtab, reader, stealth-browser-mcp, teracrawl with this list)
- self-hosted AI apps (shares Krawl, Skill_Seekers, SurfSense, WebAI2API, browser-search, browser-use, fess, moli, nanobrowser, openserp, stealth-browser-mcp with this list)
- API clients (shares Stealth-Requests, curl_cffi, httpcloak, php-curl-class, primp, reverse-api-engineer, wreq, wreq-js, wreq-python with this list)
- AI agent frameworks (shares Douyin_TikTok_Download_API, ai-website-cloner-template, browser-use, crawl4ai, moli, nanobrowser with this list)
- MCP servers (shares Douyin_TikTok_Download_API, SeleniumBase, Skill_Seekers, brightdata-mcp, firecrawl-mcp-server, stealth-browser-mcp with this list)
- search engines (shares fess, openserp with this list)
- low-code platforms (shares skills with this list)
- speech tools (shares Douyin_TikTok_Download_API with this list)
- static site generators (shares siteone-crawler with this list)
- vector databases (shares chatWeb with this list)
- API gateways
- CI/CD tools
- CMS platforms
- LLM gateways
- VPN tools
- backup tools
- code editors
- open source coding agents
- Open source databases, installed and queried
- developer CLI tools
- e-commerce platforms
- ebook readers
- game engines
- open source games
- home automation tools
- map tools
- message queues
- Open source music servers, installed and played
- observability tools
- Open source office suites, installed and launched
- self-hosted password managers
- project management tools
- screen recorders
- self-hosted analytics
- uptime monitors
- open source video editors
- video players
- whiteboard tools
- wikis
- workflow automation tools
- self-hosted Notion alternatives
- self-hosted dashboards
- self-hosted git servers