Building an Undetectable Web Scraper with CloakBrowser

Overview

Web scraping at scale faces a fundamental tension: the more requests you make, the more likely anti-bot systems will detect and block your scraper. Modern defenses like Cloudflare Turnstile, DataDome, and Akamai analyze not just IP reputation but also browser fingerprint consistency, behavioral patterns, and geographic coherence. A scraper that connects from a London IP but reports a Los Angeles timezone, or that uses incognito mode with no browsing history, is immediately flagged.

Building a truly undetectable scraper requires solving four problems simultaneously:

Fingerprint stealth -- eliminating automation signals at the browser engine level
IP reputation -- routing traffic through residential or ISP proxies that anti-bot systems trust
Geographic coherence -- aligning timezone, locale, and language with the proxy IP location
Session authenticity -- maintaining persistent browsing context (cookies, localStorage, cache) across requests to avoid incognito detection

CloakBrowser solves all four with a single Python package. Its 66 C++ source-level patches (see the C++ Patch Deep Dive) eliminate automation signals at the Chromium engine level. Combined with its proxy, GeoIP, and persistent context features, CloakBrowser provides a complete infrastructure for production-grade undetectable scraping.

This guide walks through each component and builds toward a full production scraper deployed on Docker.

Prerequisites

Install the CloakBrowser Python package with GeoIP support:

pip install cloakbrowser[geoip]

On first import, CloakBrowser automatically downloads and caches the ~200MB stealth Chromium binary. You can verify the installation:

from cloakbrowser import launch_async
import asyncio
 
async def check():
    browser = await launch_async(headless=True)
    page = await browser.new_page()
    await page.goto("https://httpbin.org/anything")
    data = await page.evaluate("""
        Object.assign({}, {
            userAgent: navigator.userAgent,
            webdriver: navigator.webdriver,
            platform: navigator.platform,
        })
    """)
    print(f"CloakBrowser ready — webdriver={data['webdriver']}")
    await browser.close()
 
asyncio.run(check())

Expected output: CloakBrowser ready — webdriver=false

Proxy Configuration

CloakBrowser supports HTTP, HTTPS, and SOCKS5 proxies via the proxy parameter in launch() / launch_async().

Basic Proxy with Inline Credentials

from cloakbrowser import launch_async
 
browser = await launch_async(
    headless=True,
    proxy="http://user:pass@proxy.16yun.cn:8888",
)
page = await browser.new_page()
await page.goto("https://httpbin.org/ip")
body = await page.text("body")
print(body)  # Should show the proxy IP, not your server IP

SOCKS5 Proxy

SOCKS5 proxies are supported using the same URI format:

from cloakbrowser import launch_async
 
browser = await launch_async(
    headless=True,
    proxy="socks5://user:pass@proxy.16yun.cn:8888",
)

Proxy Rotating Across Requests

For production scraping, rotate proxies per request or per session. The recommended pattern is to create one browser instance per proxy:

import asyncio
from cloakbrowser import launch_async
 
PROXY_LIST = [
    "http://user1:pass1@proxy.16yun.cn:8888",
    "http://user2:pass2@proxy.16yun.cn:8888",
    "http://user3:pass3@proxy.16yun.cn:8888",
]
 
async def scrape_with_proxy(url, proxy):
    """Create a CloakBrowser instance with a specific proxy and scrape one URL."""
    browser = await launch_async(headless=True, proxy=proxy)
    page = await browser.new_page()
    await page.goto(url, timeout=30000)
    content = await page.content()
    await browser.close()
    return content
 
async def main():
    urls = [
        "https://httpbin.org/anything",
        "https://httpbin.org/anything",
        "https://httpbin.org/anything",
    ]
    tasks = [
        scrape_with_proxy(url, PROXY_LIST[i % len(PROXY_LIST)])
        for i, url in enumerate(urls)
    ]
    results = await asyncio.gather(*tasks)
    for i, html in enumerate(results):
        print(f"Request {i}: {len(html)} bytes")
 
if __name__ == "__main__":
    asyncio.run(main())

GeoIP Integration: Automatic Timezone and Locale

One of the most common detection signals is a mismatch between the proxy IP's geographic location and the browser's reported timezone, locale, and language. If your proxy IP is in London but the browser reports America/New_York timezone, anti-bot systems flag the inconsistency.

CloakBrowser's GeoIP feature (geoip=True) resolves this automatically:

from cloakbrowser import launch_async
 
browser = await launch_async(
    headless=True,
    proxy="http://user:pass@proxy.16yun.cn:8888",
    geoip=True,  # Auto-detect timezone, locale, and language from proxy IP
)

When geoip=True is set, CloakBrowser performs the following at launch time:

IP geolocation: Resolves the proxy IP address to a geographic location via the geoip extras package
Timezone alignment: Sets Intl.DateTimeFormat().resolvedOptions().timeZone to the detected timezone (e.g., Europe/London)
Locale and language: Configures navigator.language, navigator.languages, and Accept-Language header based on the detected region
System time offset: Adjusts JavaScript Date behavior to match the target timezone

GeoIP Verification

Verify that GeoIP aligns your browser fingerprint with your proxy location:

async def verify_geo_alignment(browser):
    page = await browser.new_page()
    await page.goto("https://httpbin.org/anything")
    info = await page.evaluate("""
        Object.assign({}, {
            timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
            language: navigator.language,
            languages: navigator.languages,
            platform: navigator.platform,
        })
    """)
    print(f"Timezone: {info['timezone']}")
    print(f"Language: {info['language']}")
    print(f"Languages: {info['languages']}")
    return info

When using a UK proxy with geoip=True, you should see Europe/London timezone and en-GB as the primary language.

Installing GeoIP Support

The geoip extras require the geoip2 package:

pip install cloakbrowser[geoip]

This downloads the GeoLite2 database for offline IP-to-location mapping. No external API calls are made during scraping -- all lookups happen locally.

WebRTC IP Spoofing

WebRTC is a notorious source of IP leaks in automated browsers. Even when a proxy is configured, WebRTC's RTCPeerConnection can expose the client's real public IP address directly to JavaScript running on the target page -- completely bypassing the proxy.

CloakBrowser addresses this at the C++ level (see the C++ Patch Deep Dive for WebRTC patch details). The GetLocalAddresses patch filters local IP addresses during connection establishment, and when a proxy is configured, it returns the proxy IP as the public address.

You can verify WebRTC leak protection:

async def check_webrtc_leak(browser):
    page = await browser.new_page()
    result = await page.evaluate("""
        new Promise((resolve) => {
            const pc = new RTCPeerConnection({
                iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
            });
            pc.createDataChannel('');
            pc.createOffer().then((offer) => {
                const ipRegex = /\\b(?:[0-9]{1,3}\\.){3}[0-9]{1,3}\\b/g;
                const ips = [...offer.sdp.matchAll(ipRegex)]
                    .map(m => m[0])
                    .filter((v, i, a) => a.indexOf(v) === i);
                resolve(ips);
            });
        })
    """)
    print(f"WebRTC detected IPs: {result}")
    # With proxy configured, should return the proxy IP, not your server IP

Persistent Context: Avoiding Incognito Detection

By default, CloakBrowser launches in a temporary context similar to incognito mode. Many anti-bot systems check for incognito detection flags -- a browser with no cookies, no localStorage, no browsing history, and no cache appears suspicious.

CloakBrowser provides launch_persistent_context() to address this (see Issue #320 for community discussion):

from cloakbrowser import launch_persistent_context
 
browser = await launch_persistent_context(
    user_data_dir="./cloakbrowser_profile",
    headless=True,
    proxy="http://user:pass@proxy.16yun.cn:8888",
    geoip=True,
)
page = await browser.new_page()

The persistent context creates a real user profile on disk that includes:

Cookies: Persist across sessions, enabling logged-in scraping without re-authentication
localStorage and sessionStorage: Maintain application state
IndexedDB and WebSQL: Preserve client-side databases
Cache and Service Workers: Maintain cached resources and registered workers
Browsing history: Build up a realistic history over time

First-Run Profile Initialization

On the first run with a new user_data_dir, the profile is empty. For production use, pre-populate the profile with realistic data:

import asyncio
from cloakbrowser import launch_persistent_context
 
async def initialize_profile(user_data_dir, proxy_url):
    """Prime a new profile by visiting common sites to build realistic history."""
    browser = await launch_persistent_context(
        user_data_dir=user_data_dir,
        headless=False,
        proxy=proxy_url,
        geoip=True,
    )
    page = await browser.new_page()
 
    # Warm-up URLs: replace with real popular sites in your target region
    # (e.g., amazon.co.uk for UK e-commerce, bbc.co.uk for news) to build a credible history
    seed_urls = [
        "https://example.16yun.cn",
        "https://www.example.16yun.cn",
        "https://news.example.16yun.cn",
        "https://app.example.16yun.cn",
    ]
    for url in seed_urls:
        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=15000)
            await asyncio.sleep(1)
        except Exception:
            pass
 
    await browser.close()
    print(f"Profile initialized at {user_data_dir}")

Humanize: Realistic Interaction Patterns

The humanize=True flag adds three layers of behavioral realism (see the Humanize Feature Deep Dive article for source-level analysis):

from cloakbrowser import launch_async
 
browser = await launch_async(
    headless=False,
    proxy="http://user:pass@proxy.16yun.cn:8888",
    geoip=True,
    humanize=True,
)
page = await browser.new_page()
await page.goto("https://example.com")
 
# All subsequent interactions are humanized automatically
await page.click("button#submit")     # Bezier curve mouse movement
await page.fill("input#search", "cloakbrowser")  # Per-character typing
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")  # Realistic scroll

Humanize Presets

CloakBrowser offers two presets:

default (the default when humanize=True): Balanced speed and naturalness
careful: Slower movements, longer pauses between actions -- suitable for high-security targets

from cloakbrowser import launch_async
 
browser = await launch_async(
    headless=False,
    humanize=True,
    human_config={"preset": "careful"},
)

Complete Production Scraper

Combining all the components into a production-grade scraper:

import asyncio
import json
import logging
from dataclasses import dataclass, field
from typing import Optional
from cloakbrowser import launch_async, launch_persistent_context
 
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
 
@dataclass
class ScraperConfig:
    """Configuration for a CloakBrowser-based undetectable scraper."""
    proxy: str
    user_data_dir: Optional[str] = None
    headless: bool = True
    geoip: bool = True
    humanize: bool = True
    human_config: dict = field(default_factory=lambda: {"preset": "default"})
    timeout: int = 30000
 
class UndetectableScraper:
    """Production-grade undetectable web scraper using CloakBrowser."""
 
    def __init__(self, config: ScraperConfig):
        self.config = config
        self.browser = None
 
    async def __aenter__(self):
        launch_fn = (
            launch_persistent_context
            if self.config.user_data_dir
            else launch_async
        )
        kwargs = {
            "headless": self.config.headless,
            "proxy": self.config.proxy,
            "geoip": self.config.geoip,
        }
        if self.config.humanize:
            kwargs["humanize"] = True
            kwargs["human_config"] = self.config.human_config
        if self.config.user_data_dir:
            kwargs["user_data_dir"] = self.config.user_data_dir
 
        self.browser = await launch_fn(**kwargs)
        logger.info(
            f"Browser launched — proxy={self.config.proxy[:40]}... "
            f"geoip={self.config.geoip} "
            f"humanize={self.config.humanize}"
        )
        return self
 
    async def __aexit__(self, *args):
        if self.browser:
            await self.browser.close()
 
    async def scrape(self, url: str) -> dict:
        """Scrape a URL and return structured data."""
        page = await self.browser.new_page()
        try:
            await page.goto(url, timeout=self.config.timeout)
            await page.wait_for_load_state("networkidle")
 
            result = await page.evaluate("""
                Object.assign({}, {
                    title: document.title,
                    text: document.body.innerText.substring(0, 5000),
                    html_length: document.documentElement.outerHTML.length,
                    cookies: document.cookie,
                })
            """)
            result["url"] = url
            result["status"] = "success"
            return result
        except Exception as e:
            logger.error(f"Failed to scrape {url}: {e}")
            return {"url": url, "status": "error", "error": str(e)}
        finally:
            await page.close()
 
async def main():
    config = ScraperConfig(
        proxy="http://user:pass@proxy.16yun.cn:8888",
        user_data_dir="./profiles/scraper1",
        headless=True,
        geoip=True,
        humanize=True,
        human_config={"preset": "careful"},
    )
 
    async with UndetectableScraper(config) as scraper:
        result = await scraper.scrape("https://httpbin.org/anything")
        print(json.dumps(result, indent=2, ensure_ascii=False))
 
if __name__ == "__main__":
    asyncio.run(main())

Docker Deployment

For production deployment on Linux servers, Docker provides a consistent environment. CloakBrowser provides an official Docker image and a Dockerfile for custom builds (see the repository at CloakHQ/CloakBrowser).

Using the Official Docker Image

FROM cloakhq/cloakbrowser:latest
 
# Install Python dependencies for your scraper
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
 
# Copy scraper code
COPY scraper.py .
 
# Default command
CMD ["python", "scraper.py"]

Build and run:

docker build -t undetectable-scraper .
docker run --rm \
  -e PROXY_URL="http://user:pass@proxy.16yun.cn:8888" \
  undetectable-scraper

Font Installation for Headless Linux

Headless Linux environments (especially minimal Docker images) contain very few system fonts. This creates a detectable fingerprint difference because real browsers have dozens of fonts available. CloakBrowser includes font simulation at the C++ level (see C++ Patch Deep Dive font section), but for maximum authenticity, install common fonts in your Docker image:

FROM cloakhq/cloakbrowser:latest
 
# Install common fonts to match real desktop environments
RUN apt-get update && apt-get install -y \
    fonts-liberation \
    fonts-noto \
    fonts-noto-cjk \
    fonts-noto-color-emoji \
    fonts-freefont-ttf \
    fonts-dejavu-core \
    fonts-dejavu-extra \
    && rm -rf /var/lib/apt/lists/*

Docker Compose with Proxy Sidecar

For production scraping pipelines, run the proxy provider alongside CloakBrowser:

version: "3.8"
services:
  scraper:
    build: .
    environment:
      - PROXY_URL=http://proxy-service:8080
    depends_on:
      - proxy-service
    volumes:
      - ./profiles:/app/profiles
      - ./data:/app/data
 
  proxy-service:
    image: your-proxy-provider-image:latest
    ports:
      - "8080:8080"

Docker Runtime Considerations

When running CloakBrowser inside Docker:

docker run --rm \
  --cap-add=SYS_ADMIN \
  --shm-size=2gb \
  -e PROXY_URL="http://user:pass@proxy.16yun.cn:8888" \
  -v $(pwd)/profiles:/app/profiles \
  undetectable-scraper

Key arguments:

--cap-add=SYS_ADMIN: Required when running Chromium with sandbox in Docker
--shm-size=2gb: Prevents /dev/shm exhaustion during concurrent page loads
Volume mounts: Preserve persistent profiles and output data across container restarts

Best Practices

1. Match Proxy Geography to Target Site

If you are scraping a UK-only e-commerce site, use UK residential proxies with geoip=True to align timezone and locale. A London IP with America/Chicago timezone is a strong detection signal.

2. Warm Up Persistent Profiles Before Production

New profiles (empty user data directories) lack browsing history, cookies, and cache. Initialize profiles by visiting common seed URLs before production use.

3. Respect Rate Limits with Exponential Backoff

Even with perfect fingerprints and proxies, aggressive request patterns trigger behavioral detection.

import asyncio
import random
 
async def scrape_with_backoff(scraper, url, max_retries=3):
    for attempt in range(max_retries):
        result = await scraper.scrape(url)
        if result["status"] == "success":
            return result
        wait = (2 ** attempt) + random.uniform(0, 1)
        logger.warning(f"Attempt {attempt + 1} failed, retrying in {wait:.1f}s...")
        await asyncio.sleep(wait)
    return result

4. Rotate Between Multiple Persistent Profiles

Using the same profile for thousands of requests creates a detectable usage pattern. Maintain a pool of profiles and rotate them.

5. Monitor Detection Signals

CloakBrowser's patches are effective, but anti-bot systems evolve. Periodically verify your setup using detection test sites.

Common Pitfalls

Pitfall	Symptom	Fix
Proxy not applied to CDP connections	WebRTC leaks real IP	Ensure proxy set before browser launch
GeoIP database outdated	Wrong timezone for proxy IP	Run `pip install --upgrade geoip2` monthly
Persistent profile from different proxy region	Cookie/locale mismatch with current IP	Create separate profiles per geographic region
Headless font detection	Font enumeration returns 0 or very few fonts	Install fonts in Docker image

Conclusion

CloakBrowser provides all the building blocks for production-grade undetectable web scraping: C++-level fingerprint patches, proxy integration with GeoIP alignment, WebRTC leak protection, persistent browsing contexts, and humanized interaction patterns. Combined with Docker deployment, you can build a complete undetectable scraping infrastructure that reliably bypasses Cloudflare Turnstile, reCAPTCHA v3, DataDome, and other modern anti-bot systems.

The key to long-term undetectability is not any single feature but the combination -- proxy + GeoIP + persistent context + humanize -- working together to present a consistent, authentic browser identity across every request.

This article is based on CloakBrowser v0.4.10 (July 2026). Proxy and GeoIP configurations may differ between Free (Chromium 146) and Pro (Chromium 148) tiers.