anishfyi.com Projects All writings

July 28, 202612 min readOpen Source, Python, Web Scraping, curl_reap

Why your scraper gets a 403 and your browser does not

The block usually happens in the TLS handshake, before a single HTTP header is read. On JA3 fingerprints, self-healing selectors, and why I hand-wrote a crawl engine instead of importing Scrapy.

You open a page in a browser and it loads. You fetch the same URL from Python and get a 403. So you copy the User-Agent from devtools. Still 403. You copy every header, in order, including Accept-Language and Sec-Fetch-Mode. Still 403.

At this point most people conclude the site is doing something clever with JavaScript and reach for a headless browser, which costs a few hundred megabytes of Chromium and roughly two orders of magnitude more time per page.

Usually that is the wrong diagnosis. You were blocked before you sent a single HTTP header.

The handshake gives you away

Before any HTTP happens, your client and the server negotiate TLS. Your client sends a ClientHello that announces, in a specific order, which TLS version it supports, which cipher suites it will accept, which elliptic curves, which extensions, which signature algorithms, and which ALPN protocols.

None of that is secret and none of it is spoofed by setting a header. It is a property of the TLS library underneath your HTTP client. And the exact combination and ordering differs between OpenSSL as Python's requests uses it, BoringSSL as Chrome uses it, and Apple's Secure Transport as Safari uses it.

Hash that combination and you get a short, stable fingerprint. That is JA3, and its successor JA4. A CDN can compute it on every connection at negligible cost, and the result answers a question that headers cannot: what actually made this request?

So when your carefully header-matched Python client announces a Chrome User-Agent while presenting an OpenSSL fingerprint, you have not disguised yourself. You have made yourself more interesting. A default-configured Python client is at least honest. A Python client wearing a Chrome name badge is a rule that writes itself.

Presenting a real fingerprint

The fix is not to fake the fingerprint at the header level. It is to actually perform the handshake the way the browser performs it. curl_cffi does this by binding to a curl build patched to reproduce specific browsers' TLS stacks, which is why it is the foundation curl_reap sits on. 

import curl_reap as reap

page = reap.get("https://quotes.toscrape.com", impersonate="chrome124")
print(page.css("span.text::text").getall())
print(page.css_first("small.author::text"))

That impersonate argument is the whole trick. The request goes out with a genuine Chrome 124 ClientHello, so the fingerprint check passes and the request is evaluated on its behaviour rather than its stack. The parsing on the next line works like parsel, because once you are through the door you still have a document to read.

Worth saying plainly: this defeats fingerprint-based classification, not rate limiting, not authentication, not a paywall, and not a site's terms. It gets a well-behaved crawler treated as well-behaved instead of being rejected for the library it was written in.

The second thing that breaks: the markup moved

Your scraper now works. Six weeks later it returns empty lists, because someone renamed a.buy-btn to a.purchase-cta.

Nothing was wrong with your logic. The element is still there, still in the same position in the tree, still surrounded by the same siblings, still carrying the same kind of content. Only the label changed. But a CSS selector is a string match, so a rename is indistinguishable from a deletion.

Scrapling's insight was that you can identify an element by its structural signature rather than its name, and re-find it when the name stops matching. curl_reap implements the same idea: 

page = reap.get("https://shop.example.com/item/42")
page.css_first("a.buy-btn").save("buy_button")     # remember its shape

# weeks later, the class is now "purchase-cta" and the old selector misses:
later = reap.get("https://shop.example.com/item/99")
btn = later.css_first("a.buy-btn", auto_match=True, identifier="buy_button")
print(btn.attr("href"))                            # found anyway

You save the element once. On a later run, when the selector misses, auto_match relocates it by the signature you saved. There are also page.find_by_text("Sign in") and page.find_similar(element) for the cases where you never had a stable selector to begin with.

This is not magic and it should not be your first line of defence. It is a degradation path. The alternative is a scraper that returns an empty list and a pipeline that cheerfully writes zero rows, which is the worst possible failure mode: silent, and shaped exactly like a real answer.

The third thing: one page is not a crawl

Fetching one URL well is a solved problem. Fetching fifty thousand is a different discipline: concurrency, per-domain politeness, retries with backoff, deduplication, depth and domain limits, and somewhere to put the results. That is what Scrapy is for, and Scrapy is genuinely good at it.

The trouble is that these three problems have three different owners. curl_cffi gets you through the door but has no parser and no crawl engine. Scrapling survives markup drift but does not crawl. Scrapy crawls but presents an OpenSSL handshake, so on a fingerprinting site it never gets to demonstrate any of it.

So a real scraping project ends up gluing three libraries with three configuration models and three failure vocabularies, and the glue is where the bugs live.

curl_reap takes the best idea from each and puts them behind one API. The crawl engine is hand-written rather than a Scrapy dependency. That was a deliberate choice after reading all three side by side: importing Scrapy would have meant importing its scheduler, its settings system, its signals, its item pipelines and its project layout, in order to use one of those things.

The scheduler is continuous rather than batched, which matters more than it sounds. A batched scheduler waits for a wave of requests to finish before starting the next, so one slow page holds the whole crawl at the speed of its worst response. A continuous scheduler keeps every worker fed. Per-domain AutoThrottle adapts pace to observed latency, and dedup, depth limits, domain limits, retries and pipelines are all in the same place.

What a scrape actually wants

Here is a thing I did not expect to matter as much as it does. Most scrapes do not want a bespoke selector chain. They want one of about six things, and those six things are structurally identical on almost every site because SEO made them so. 

page = reap.get("https://example.com/product/42")

page.jsonld()      # all JSON-LD blocks as dicts (product/article/event data)
page.meta_tags()   # {title, description, og:*, twitter:*, canonical, ...}
page.links(internal_only=True)   # absolute urls with their anchor text
page.images()      # url + alt, handling lazy data-src
page.tables()      # every table as list-of-rows
page.markdown()    # readable page content as markdown

jsonld() is the one people skip and should not. A product page that fights your selectors will very often hand you a complete, structured, schema.org Product object in a script tag, because Google asked for it. The site is already publishing clean data. Read that instead of parsing the rendering of it.

markdown() exists for a newer reason: it is the right shape to feed a model. Handing an LLM raw HTML spends most of your tokens on div soup and class names. Handing it markdown spends them on the content.

Resilience, without writing it again

# smart retries with exponential backoff + jitter, honoring Retry-After
s = reap.Session(retry_policy=reap.RetryPolicy(retries=4, backoff=0.5))

# rotate real browser fingerprints and proxies across requests
s = reap.Session(rotate="random", proxy=["http://p1:8080", "http://p2:8080"])

# disk cache: repeat GETs come from disk, which is what dev loops actually need
s = reap.Session(cache=reap.DiskCache(ttl=3600))
r = s.get(url); r2 = s.get(url)   # r2.from_cache is True

The disk cache is the one I use most, and not for production. Developing a scraper means running the same fetch forty times while you get the selectors right. Without a cache you are hammering someone else's server to debug your own code, which is both rude and slow.

The recent work is all about being honest

Version 0.3.0 is a good illustration of what maintaining a scraper actually consists of, and none of it is glamorous:

The honest limits

curl_reap does not execute JavaScript. If a page renders its content client-side from an XHR, the right move is usually to find that XHR and call it directly, which is faster and more stable than rendering anyway. When it genuinely is not, you want a browser and curl_reap is the wrong tool.

It does not solve CAPTCHAs, it does not defeat behavioural analysis that scores mouse movement and timing, and it will not make an unwelcome crawler welcome. It requires Python 3.9 or newer and pulls in exactly three runtime dependencies: curl_cffi, lxml and cssselect. That small dependency set is a feature, and keeping it small is a constraint I enforce on every change.

pip install curl_reap. MIT licensed, sponsored by NodeMaven. Source at github.com/anishfyi/curl_reap, full API reference at anishfyi.github.io/curl_reap.

Working on something in this territory? Book thirty minutes.