Your scraper worked fine yesterday. Today it's returning 403s on every request. You've been blocked — and you're not sure why.
This happens because anti-bot systems in 2026 are layered: they stack IP reputation checks, TLS fingerprinting, browser behavior analysis, and CAPTCHA triggers into a detection pipeline that most scrapers trip at multiple points simultaneously. Fixing one layer while ignoring the others is why most "solutions" fail within hours.
This guide covers every layer, in the order they matter. By the end, you'll understand exactly what's getting you blocked and how to fix it.
The first thing any request hits is an IP reputation check. Before your request is even routed to the application, the edge layer — Cloudflare, Akamai, AWS Shield, or a homegrown WAF — looks up your IP against known blocklists, ASN databases, and behavioral history.
What gets flagged: - IPs in commercial data center ASNs (AWS, Hetzner, DigitalOcean, etc.) - IPs that have appeared in prior bot reports - IPs making requests at machine-speed from a single source - Entire subnets that have been flagged by honeypots
The fix: proxy rotation with clean IPs
Rotating your IP address across requests is the foundational anti-blocking technique. But not all IP pools are equal.
Free proxy lists are useless — every IP on them has been hammered to death and appears on every major blocklist. Shared commercial datacenter IPs are better, but still carry the ASN risk from being hosted on identifiable server infrastructure.
For protected targets, residential proxies are the only reliable solution at the IP layer. These are IP addresses assigned by consumer ISPs to real homes — the same class of IPs your browser uses when you're on home Wi-Fi. Anti-bot systems treat them with much lower suspicion because blocking residential IP ranges risks blocking legitimate customers.
NinjaProxy maintains a pool of 550,000+ IPs across 30+ data centers and residential networks. The key advantage is fresh IPs — not overworked addresses already flagged by other users. Overused shared pools are one of the most common reasons proxy solutions stop working even when rotation is in place.
Rotation strategy:
Don't rotate on every request. Many anti-bot systems detect "teleporting" users — an IP in New York on request 1, Tokyo on request 2 — as a bot signal. Use sticky sessions: one IP handles a complete user journey (load page → wait → click → extract), then rotate to the next journey.
For high-concurrency scraping, a pool of 50–100 residential IPs rotated at the session level outperforms 500 IPs rotated per-request.
An IP check is just the entry gate. Modern anti-bot systems also fingerprint your HTTP requests — analyzing headers, TLS settings, and HTTP/2 frame ordering to determine whether the request looks like a real browser or a script.
What gets flagged:
- Missing or inconsistent headers (Accept, Accept-Language, Accept-Encoding, sec-ch-ua, sec-fetch-*)
- TLS fingerprints that don't match the claimed browser (your User-Agent says Chrome 124 but your TLS handshake looks like Python's requests library)
- HTTP/2 settings that diverge from real browser behavior
- User-Agent strings that are outdated (Chrome versions from 2022 are statistically implausible in 2026 real traffic)
The fix: align every signal
The most common mistake is rotating User-Agent headers in isolation while leaving everything else static. If your User-Agent claims to be Chrome on macOS but your Accept-Language header says en-US,en;q=0.5 (Firefox's default), the mismatch is a bot signal.
A consistent browser profile means:
- User-Agent matches current browser versions (check current stable release numbers quarterly)
- Accept, Accept-Language, Accept-Encoding match what that browser actually sends
- sec-ch-ua client hints match the browser version in User-Agent
- sec-fetch-site, sec-fetch-mode, sec-fetch-dest reflect realistic navigation context
For TLS fingerprinting specifically, the requests and httpx libraries in Python have distinct TLS fingerprints that experienced detection systems identify immediately. Tools like curl-impersonate or browser automation (Playwright, Puppeteer with stealth mode) solve this by replaying genuine browser TLS handshakes.
If you're using a headless browser (Playwright, Puppeteer, Selenium), the detection system looks past the request headers and into the JavaScript environment. Browser fingerprinting checks:
navigator.webdriver — set to true in every unpatched headless instancenavigator.plugins — empty arrays are a strong bot signalThe fix: stealth automation
For Playwright, playwright-extra with the stealth plugin patches the most obvious headless tells. For Puppeteer, puppeteer-extra-plugin-stealth does the same. Neither is bulletproof against the most sophisticated detection (Cloudflare's Managed Challenge, PerimeterX's advanced fingerprinting), but they eliminate the low-hanging-fruit detections that catch most vanilla headless browsers.
For the hardest targets, pre-rendered browser profiles — real Chrome instances with established cookie jars, browsing history, and fingerprint consistency — are more reliable than stealth plugins, but require significantly more infrastructure.
Behavioral detection is the hardest layer to defeat because it watches how your scraper moves through a site over time, not just how a single request looks.
What gets flagged: - Request rates far above human reading speed (3 pages/second is obviously not human) - Zero variance in timing (machine-perfect intervals between requests) - Navigation patterns that skip pages real users would visit (going directly to product data pages without hitting homepage or category pages) - Zero interaction signals: no mouse events, no scroll, no focus/blur - Identical session paths across multiple IPs
The fix: throttle like a human
Add delays between requests. Not fixed delays — randomized delays. A human reading a page takes 5–30 seconds before clicking through. A bot requesting 3 pages/second is obvious. A bot requesting 1 page every 7–14 seconds (with variance) looks far more plausible.
import time
import random
def human_delay(min_seconds=3, max_seconds=12):
"""Randomized delay to simulate human reading time."""
time.sleep(random.uniform(min_seconds, max_seconds))Beyond timing, vary your navigation paths. Real users don't follow the same path through a site on every visit. Occasionally visit category pages, follow internal links, and leave some pages without extracting data — just as a real user would bounce around.
Even well-behaved scrapers hit rate limits when they don't account for per-IP or per-session request ceilings. Rate limiting operates independently of bot detection — it's a server resource protection mechanism.
What triggers rate limits: - Exceeding per-minute or per-hour request thresholds from one IP - Sustained high-volume requests across a session - Too many concurrent connections from one source
The fix: stay under the limit and rotate before you hit it
Monitor your request cadence. Build in automatic rotation triggers: if you've made N requests from one IP within a time window, retire that IP and pull a fresh one from your pool before the rate limiter fires.
class ThrottledScraper:
def __init__(self, proxy_pool, max_requests_per_ip=50):
self.pool = proxy_pool
self.max_requests = max_requests_per_ip
self.current_proxy = self.pool.get_next()
self.request_count = 0
def get_proxy(self):
if self.request_count >= self.max_requests:
self.current_proxy = self.pool.get_next()
self.request_count = 0
self.request_count += 1
return self.current_proxyWith NinjaProxy's unlimited bandwidth model, rotating across IPs doesn't create unexpected per-GB costs — you can rotate aggressively without watching a usage meter.
CAPTCHAs are a response to failed detection, not first-line detection. If your scraper reaches a CAPTCHA, something in the earlier layers flagged it.
The right approach is avoidance, not solving.
CAPTCHA solving services exist (2Captcha, CapMonster, Anti-Captcha), but they're slow (3–15 seconds per solve), expensive at scale, and increasingly ineffective against v3/invisible variants that don't present a UI challenge at all.
Avoidance means fixing the upstream signals so you never trigger the CAPTCHA in the first place: 1. Clean IP reputation (residential proxies) 2. Correct request fingerprinting 3. Patched browser fingerprinting 4. Human-paced behavioral signals
If you're consistently hitting CAPTCHAs despite fixing these, the target is running a challenge mode (Cloudflare "I'm Under Attack" or Managed Challenge) that's aggressive by policy, not by detection. At this point, mobile proxies — 4G/5G IPs on carrier networks — are often the only path through, because cellular IPs carry the highest trust scores of any IP type.
Some sites embed hidden links or fields — invisible to real users but visible to scrapers that follow every link or submit every form. Visiting a honeypot link immediately flags your session.
The fix: check display:none, visibility:hidden, and opacity:0 on every link before following it. If you're using browser automation, only interact with elements that are actually visible.
Here's what a robust 2026 scraping setup looks like end-to-end:
| Layer | Problem | Solution |
|---|---|---|
| IP reputation | Commercial ASN flagged | Residential proxy rotation (sticky sessions) |
| Request fingerprinting | Headers/TLS mismatch | Full browser profile alignment |
| Browser fingerprinting | Headless tells | Stealth plugins or real browser instances |
| Behavioral analysis | Machine-speed patterns | Randomized delays, varied navigation |
| Rate limiting | Per-IP request ceiling | Proactive rotation before threshold |
| CAPTCHA | Triggered by upstream flags | Fix upstream; use mobile proxies for hard targets |
| Honeypots | Invisible link traps | Skip hidden elements |
You don't need every layer for every target. Match your stack to the site's actual protection level:
requests + any datacenter IPsThe proxy layer is where most scraping operations fail, because it's the one infrastructure choice that affects every detection layer simultaneously.
For accessible targets: NinjaProxy private datacenter proxies at $1.72/proxy give you dedicated, non-shared IPs that haven't been flagged by prior users. Shared proxies ($0.09/proxy) work for high-volume targets where occasional blocks are acceptable.
For protected targets: NinjaProxy residential proxies at $7.75/proxy. The unlimited bandwidth model means you can run high-volume residential scraping without per-GB overage costs eating into the economics.
For the hardest targets: NinjaProxy's 4G/5G mobile proxies — carrier-grade IPs with the highest trust scores available.
The key factor across all tiers is IP freshness. Overused pools — especially the cheap "unlimited" residential pools that recycle IPs aggressively — burn through their addresses' reputation quickly. NinjaProxy's pool management prioritizes clean, uncontested IPs that haven't been saturated by other users' traffic.
403 Forbidden immediately: IP reputation issue. Switch to residential proxies.
200 OK but served a CAPTCHA page: Fingerprinting issue. Align request headers; consider stealth browser.
Works for N requests then fails: Rate limiting. Add delays and proactive IP rotation.
Works fine then fails with redirect to /blocked or /security-check: Behavioral detection. Reduce speed, add variance, vary navigation paths.
Works with residential IPs but not datacenter: ASN-level block. Stay on residential; consider ISP or mobile proxies for persistent issues.
Getting scraping right is mostly an infrastructure problem, not a code problem. The same Python scraper that fails with datacenter IPs succeeds immediately with residential proxies, because the detection systems ruling it out aren't looking at your code — they're looking at where your requests come from.
NinjaProxy has been running proxy infrastructure since 2007 with 550,000+ IPs, 30+ data centers, unlimited bandwidth, and 99.999% uptime. Whether you're starting with shared datacenter proxies or scaling to residential and mobile, the full suite is available from one provider.
View proxy plans and pricing →
*Related reading:*