web-crawler
Tested projects carrying the web-crawler topic, with their current verdicts. Topics come from each project's own metadata and Argusic's pool tags. Spellings of the same topic are gathered here under one name, so this page holds every project tagged any of them.
Measured by Argusic on a fresh machine every time. Every number links to its evidence.
What Argusic measured in web-crawler
Argusic installed and ran 18 web-crawler projects on clean machines: 13 reached a running state, 5 only ran against stand-in services. Last tested November 4, 2026.
Every tested web-crawler project
| subject | verdict | language | runs | last tested |
|---|---|---|---|---|
| crawlee-python | Runs | Python | 2 | |
| crawlab | Runs | Go | 2 | |
| crawl4ai | Runs | Python | 2 | |
| crawler | Runs | Java | 1 | |
| agent-fetch | Runs | TypeScript | 2 | |
| crawler | Runs | PHP | 1 | |
| news-crawl | Runs with mocks | Java | 1 | |
| graphlit-mcp-server | Runs with mocks | TypeScript | 1 | |
| google-news-scraper | Runs with mocks | JavaScript | 1 | |
| sparkler | Runs | Python | 1 | |
| reader | Runs | TypeScript | 1 | |
| Stealth-Requests | Runs | Python | 1 | |
| spidr | Runs with mocks | Ruby | 1 | |
| stormcrawler | Runs | Java | 1 | |
| linkinator | Runs | TypeScript | 1 | |
| spider | Runs | Rust | 1 | |
| Scrapegraph-ai | Runs with mocks | Python | 2 | |
| firecrawl-mcp-server | Runs | JavaScript | 3 |
All tested projects: subjects.