WRITING / Engineering

Why your scraper gets a 403 and your browser does not

The block usually happens in the TLS handshake, before any header is read. On handshake fingerprints, self-healing selectors and a hand-written crawl engine.

Anish Kr Singh ·

You open a page in a browser and it loads. You fetch the same address from Python and get a 403 Forbidden. So you copy the User-Agent from the browser’s developer tools. Still 403. You copy every header, in order, including Accept-Language and Sec-Fetch-Mode. Still 403.

At this point most people conclude the site is doing something clever with JavaScript and reach for a headless browser, which costs a few hundred megabytes of Chromium and roughly two orders of magnitude more time per page.

Usually that is the wrong diagnosis. You were blocked before you sent a single header.

The handshake gives you away

Before any Hypertext Transfer Protocol (HTTP) request goes out, your client and the server negotiate Transport Layer Security (TLS). Your client sends a ClientHello message that announces, in a specific order, which TLS versions it supports, which cipher suites it will accept, which elliptic curves, which extensions, which signature algorithms, and which application protocols it offers through Application-Layer Protocol Negotiation (ALPN).

None of that is secret, and none of it changes when you set a header. It is a property of the TLS library underneath your HTTP client. The exact combination and ordering differs between OpenSSL as Python’s requests uses it, BoringSSL as Chrome uses it, and Apple’s Secure Transport as Safari uses it.

Hash that combination and you get a short, stable fingerprint. That is JA3, and its successor JA4. A content delivery network (CDN) can compute it on every connection at negligible cost, and the result answers a question that headers cannot: what actually made this request?

So when your carefully header-matched Python client announces a Chrome User-Agent while presenting an OpenSSL fingerprint, you have not disguised yourself. You have made yourself more interesting. A default-configured Python client is at least honest. A Python client wearing a Chrome name badge is a rule that writes itself.

Presenting a real fingerprint

The fix is not to fake the fingerprint at the header level. It is to perform the handshake the way the browser performs it. curl_cffi does this by binding to a curl build patched to reproduce specific browsers’ TLS stacks. When I wrote this, it was the foundation curl_reap sat on. It no longer is; see the update at the end.

import curl_reap as reap

page = reap.get("https://quotes.toscrape.com", impersonate="chrome124")
print(page.css("span.text::text").getall())
print(page.css_first("small.author::text"))

That impersonate argument is the whole trick. The request goes out with a genuine Chrome 124 ClientHello, so the fingerprint check passes and the request is judged on its behaviour rather than its stack. The parsing on the next line works like the parsel library, because once you are through the door you still have a document to read.

Worth saying plainly: this defeats fingerprint-based classification. It does not defeat rate limiting, authentication, a paywall, or a site’s terms. It gets a well-behaved crawler treated as well-behaved instead of rejected for the library it was written in.

The second thing that breaks: the markup moved

Your scraper now works. Six weeks later it returns empty lists, because someone renamed a.buy-btn to a.purchase-cta.

Nothing was wrong with your logic. The element is still there, in the same position in the tree, with the same siblings and the same kind of content. Only the label changed. But a Cascading Style Sheets (CSS) selector is a string match, so a rename looks exactly like a deletion.

Scrapling’s insight was that you can identify an element by its structural signature rather than its name, and re-find it when the name stops matching. curl_reap implements the same idea:

page = reap.get("https://shop.example.com/item/42")
page.css_first("a.buy-btn").save("buy_button")     # remember its shape

# weeks later, the class is now "purchase-cta" and the old selector misses:
later = reap.get("https://shop.example.com/item/99")
btn = later.css_first("a.buy-btn", auto_match=True, identifier="buy_button")
print(btn.attr("href"))                            # found anyway

You save the element once. On a later run, when the selector misses, auto_match relocates it by the saved signature. There are also page.find_by_text("Sign in") and page.find_similar(element) for cases where you never had a stable selector.

This is not magic, and it should not be your first line of defence. It is a degradation path. The alternative is a scraper that returns an empty list and a pipeline that cheerfully writes zero rows, which is the worst failure mode: silent, and shaped exactly like a real answer.

The third thing: one page is not a crawl

Fetching one page well is a solved problem. Fetching fifty thousand is a different discipline: concurrency, per-domain politeness, retries with backoff, deduplication, depth and domain limits, and somewhere to put the results. That is what Scrapy is for, and Scrapy is genuinely good at it.

The trouble is that these three problems have three different owners. curl_cffi gets you through the door but has no parser and no crawl engine. Scrapling survives markup drift but does not crawl. Scrapy crawls but presents an OpenSSL handshake, so on a fingerprinting site it never gets to show what it can do.

So a real scraping project ends up gluing three libraries with three configuration models and three failure vocabularies, and the glue is where the bugs live.

curl_reap takes the best idea from each and puts them behind one application programming interface (API). The crawl engine is hand-written rather than a Scrapy dependency. I chose that after reading all three side by side: importing Scrapy would have meant importing its scheduler, its settings system, its signals, its item pipelines and its project layout, to use one of those things.

The scheduler is continuous rather than batched, which matters more than it sounds. A batched scheduler waits for a wave of requests to finish before starting the next, so one slow page holds the whole crawl to the speed of its worst response. A continuous scheduler keeps every worker fed. Per-domain AutoThrottle adapts the pace to observed latency, and deduplication, depth limits, domain limits, retries and pipelines all live in the same place.

What a scrape actually wants

I did not expect this to matter as much as it does. Most scrapes do not want a bespoke selector chain. They want one of about six things, and those six things are structurally identical on almost every site because search engine optimisation (SEO) made them so.

page = reap.get("https://example.com/product/42")

page.jsonld()      # all JSON-LD blocks as dicts (product/article/event data)
page.meta_tags()   # {title, description, og:*, twitter:*, canonical, ...}
page.links(internal_only=True)   # absolute urls with their anchor text
page.images()      # url + alt, handling lazy data-src
page.tables()      # every table as list-of-rows
page.markdown()    # readable page content as markdown

jsonld() is the one people skip and should not. A product page that fights your selectors will very often hand you a complete schema.org Product object as JavaScript Object Notation for Linked Data (JSON-LD) in a script tag, because Google asked for it. The site is already publishing clean data. Read that instead of parsing its rendering.

markdown() exists for a newer reason: it is the right shape to feed a model. Handing a large language model (LLM) raw HyperText Markup Language (HTML) spends most of your tokens on div soup and class names. Handing it Markdown spends them on the content.

Resilience, without writing it again

# smart retries with exponential backoff + jitter, honoring Retry-After
s = reap.Session(retry_policy=reap.RetryPolicy(retries=4, backoff=0.5))

# rotate real browser fingerprints and proxies across requests
s = reap.Session(rotate="random", proxy=["http://p1:8080", "http://p2:8080"])

# disk cache: repeat GETs come from disk, which is what dev loops actually need
s = reap.Session(cache=reap.DiskCache(ttl=3600))
r = s.get(url); r2 = s.get(url)   # r2.from_cache is True

The disk cache is the one I use most, and not in production. Developing a scraper means running the same fetch forty times while you get the selectors right. Without a cache you are hammering someone else’s server to debug your own code, which is both rude and slow.

The recent work is all about being honest

The 0.3.0 changes are a good illustration of what maintaining a scraper actually involves, and none of it is glamorous. That version was never tagged; the same changes shipped in 1.0.0.

  • Encoding detection. Servers omit the charset, or state one and send another. curl_reap now resolves it in order (the byte order mark, then Content-Type, then <meta charset>, then charset-normalizer if installed) and exposes the result on Response.encoding. Mojibake, text garbled by the wrong encoding, is a data-quality bug that survives all the way to your database.
  • Conditional cache revalidation. A stale cache entry now revalidates with If-None-Match and If-Modified-Since. A 304 Not Modified serves the cached body instead of re-downloading it. Politeness and speed happen to be the same feature here.
  • robots.txt Crawl-delay and sitemap discovery. The crawler floors its per-domain pace at the site’s declared Crawl-delay, and surfaces any sitemaps it declares. If a site tells you how fast to go and where its index is, arguing with it just makes more work.
  • Gzipped sitemaps. .xml.gz sitemaps and indexes are decompressed before parsing, because large sites ship them that way and a silent parse failure looks exactly like an empty site.
  • Auto proxy rotation on a block. On a 403, 407 or 429 with a proxy pool configured, the session rotates onto a fresh proxy and retries.

The honest limits

curl_reap’s HTTP client does not execute JavaScript. If a page renders its content client-side from an XMLHttpRequest (XHR), the right move is usually to find that request and call it directly, which is faster and more stable than rendering anyway. When it genuinely is not, you want a browser.

It does not solve Completely Automated Public Turing tests to tell Computers and Humans Apart (CAPTCHAs), it does not defeat behavioural analysis that scores mouse movement and timing, and it will not make an unwelcome crawler welcome. When I wrote this it required Python 3.9 or newer and pulled in exactly three runtime dependencies: curl_cffi, lxml and cssselect. That small dependency set was a feature, and keeping it small is a constraint I enforce on every change.

Update, October 2026

curl_reap has changed a lot since July. The parts that affect this post, from the changelog:

  • No more curl_cffi. Since 1.0.0 (22 August 2026) the transport is written in Python, with no compiled curl binding. The 2.0.0 code on the main branch has its own TLS 1.3 engine. Its browser templates were captured from curl_cffi 0.16.3, and target names such as chrome124 still work, so the first example in this post runs on 2.0.0.
  • Five dependencies instead of three. 2.0.0 needs lxml, cssselect, cryptography, hpack and brotli, all installed from prebuilt wheels. It still needs Python 3.9 or newer.
  • JavaScript rendering is an option. 1.1.0 (30 September 2026) added reap.render() through Playwright, in an optional js extra.
  • Install from git for now. 2.0.0 is not released yet, and pip install curl_reap still installs 0.2.2 from the Python Package Index (PyPI), which predates the new transport and the 0.3.0 changes above. Installation has the git command.

MIT licensed, sponsored by NodeMaven. Source at github.com/anishfyi/curl_reap, full API reference at velofy.co/curl_reap.

← All writing