GitHub - D4Vinci/Scrapling: 🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Scrapling is an open-source Python library that combines fast HTTP fetching, stealth browser automation, adaptive parsing, and concurrent crawling in one package. It can be driven from Python code, the terminal, or AI agents through an MCP server.
- Handles the full scraping stack: plain HTTP requests, JavaScript rendering, Cloudflare Turnstile handling, and background capture of the API calls behind a page.
- Spider API is Scrapy-like, with concurrency limits, AutoThrottle that adjusts per-domain delays from server response, robots.txt compliance, pause/resume, and JSON/CSV/XML/JSONL export.
- Element tracking relocates selectors after a site changes its layout, and the library can turn whole sites into clean Markdown for RAG pipelines.
Scrapling is a Python web scraping library that tries to cover the whole pipeline in one package: fetching pages over plain HTTP, driving stealth browsers when JavaScript or anti-bot checks get in the way, parsing with CSS, XPath, regex or adaptive element tracking, and running concurrent crawlers. The fetchers are the core of it. `Fetcher` handles fast HTTP requests with browser-like TLS fingerprints, `StealthyFetcher` runs Chromium through Playwright, and the session classes keep cookies and state alive across requests. It also claims to handle Cloudflare Turnstile and interstitial challenges, though that's a vendor claim you'd want to test against your own targets.
The spider API will look familiar if you've used Scrapy. You subclass `Spider`, set `start_urls`, and write an async `parse` method that yields dicts, and the framework does the rest. The features that matter for longer jobs are the ones that usually take the most effort to build yourself: per-domain throttling, an AutoThrottle that adjusts delays based on how the server responds and backs off on `Retry-After`, optional robots.txt compliance, pause and resume through a `crawldir`, and a development mode that caches responses to disk so you can iterate on `parse()` without hammering the site again. Output goes to JSON, JSONL, CSV, or XML with one method call. Templates for rule-based crawling, sitemaps, feeds, and Shopify stores cover the common starting points.
The rest of the package is aimed at people who want the scraper to be driven by something other than their own Python code. There's a CLI, an interactive shell, an MCP server so AI agents can browse and extract, and an Agent Skill. Some features are worth a closer look for anyone doing serious work: background API capture on browser sessions, which lets you read the JSON endpoints a page calls without re-requesting them, and remote browser support over CDP. The README is also heavy with sponsor banners for proxy and CAPTCHA services, so read the feature list rather than the ads, and check the project's license and maintenance activity before depending on it in production.