Node.js HTTP Client Tunnel Proxy: axios / got / http Comparison
axios, got, and native http module — three approaches to tunnel proxy integration.
Engineering Blog
Category: Technical Guide
45 posts in this category.
Python mitmproxy Tunnel Proxy: Forward to Crawler Proxy via Upstream Mode
mitmproxy upstream mode + addon script for Crawler Proxy integration.
Golang Tunnel Proxy: net/http and resty Four IP Control Scenarios
Go net/http and resty four tunnel scenarios with native HTTPS Proxy-Tunnel support.
Python Selenium Chrome Tunnel Proxy: Browser Automation with Crawler Proxy
Selenium Chrome + Crawler Proxy: Chrome extension-based proxy auth integration.
Node.js Puppeteer Tunnel Proxy: Chrome Automation with Crawler Proxy
Puppeteer + Crawler Proxy: Chrome automation proxy integration.
Node.js Playwright Tunnel Proxy: Browser Automation with Crawler Proxy
Playwright + Crawler Proxy: browser automation proxy integration.
Python Scrapy Tunnel Proxy: Four IP Control Scenarios
Scrapy middleware configuration for four proxy scenarios, including HTTP Proxy-Tunnel and HTTPS limitation notes.
Python aiohttp Tunnel Proxy: Async Four IP Control Scenarios
aiohttp async four scenarios with proxy_headers for HTTPS Proxy-Tunnel support.
Python httpx Tunnel Proxy: Four IP Control Scenarios with Async HTTP
httpx four tunnel proxy scenarios with httpx.Proxy for HTTPS Proxy-Tunnel support.
Python requests Tunnel Proxy: Four IP Control Scenarios for Crawler Proxy
requests full four-scenario implementation with custom HTTPAdapter for HTTPS Proxy-Tunnel.
Python urllib Tunnel Proxy: Standard Library Four Scenarios
Python standard library urllib implements four tunnel proxy ...
agent-browser + 16Yun Proxies: Enterprise AI Scraping Deployment
Session isolation, security policies, proxy selection, and production deployment checklist for agent-browser.
agent-browser Network Control: Request Interception & Page Diagnostics
network route to intercept requests, network har to record performance, react tree for component inspection, vitals for metrics.
Python feapder & ScrapySplash Tunnel Proxy: Framework Extensions
feapder AirSpider and Scrapy+Splash integrating 16Yun Crawle...
agent-browser Batch Automation & AI Chat
batch for multi-step execution in one call, chat for natural language browser control, concurrent sessions.
Python pyppeteer & Selenium Firefox Tunnel Proxy: Browser Automation
pyppeteer and Selenium Firefox integrating 16Yun Crawler Pro...
PHP Tunnel Proxy: curl / Guzzle / Stream Three Implementations
PHP curl, Guzzle, and Stream Context integrating 16Yun Crawl...
agent-browser Session Management: Auth Persistence & State Reuse
--profile to reuse Chrome login state, --session-name for auto persistence, state save/load for explicit management.
C++ Tunnel Proxy: libcurl and CPR Two Implementations
C++ libcurl and CPR libraries integrating 16Yun Crawler Prox...
agent-browser Snapshot & Screenshots: Make Your AI Understand the Page
snapshot -i for interactive elements only, screenshot --annotate with numbered labels, diff for change detection.
C# Tunnel Proxy: HttpClient and HttpWebRequest
C# HttpClient and HttpWebRequest integrating 16Yun Crawler P...
agent-browser 101: AI-First Browser Automation CLI
npm install -g agent-browser, one command to open a browser, snapshot to understand page structure, click/fill to interact.
Trafilatura + CloakBrowser: The Complete Full-Text Scraping Pipeline
CloakBrowser renders JS/SPA pages → Trafilatura extracts clean text. Solve the 'JS-rendered content can't be extracted' problem.
Rust Tunnel Proxy: reqwest and isahc
Rust reqwest and isahc HTTP clients integrating 16Yun Crawle...
Ruby Tunnel Proxy: Faraday and HTTParty
Ruby Faraday and HTTParty integrating 16Yun Crawler Proxy....
Trafilatura Advanced: Custom Extraction, Language Detection & Performance
Advanced Trafilatura: custom element exclusion, language detection, offline batch processing, and incremental updates.
Swift Tunnel Proxy: Alamofire and URLSession
Swift Alamofire and URLSession integrating 16Yun Crawler Pro...
Batch Scraping: Trafilatura + 16Yun Proxies in Production
From single-page extraction to million-scale batch pipelines: concurrency control, proxy rotation, error handling, and storage.
Perl LWP Tunnel Proxy
Perl LWP::UserAgent integrating 16Yun Crawler Proxy....
Precision Extraction: Content, Metadata, and Tables — Trafilatura Deep Dive
Deep dive into Trafilatura's extraction engine with benchmark data, metadata fields, and tuning strategies.
Smart Crawling: Discover and Scrape Full Websites with Trafilatura
Sitemap discovery → Feed tracking → URL management → bulk extraction — a complete full-site scraping workflow.
Node.js Advanced Tunnel Proxy: undici / superagent / https
Node.js undici, superagent, and native https module integrat...
Node.js Legacy Tunnel Proxy: phantomjs / casperjs / request
Deprecated but still widely deployed Node.js tools integrati...
Trafilatura 101: From HTML to Structured Text in 3 Lines
pip install and 3 lines of code to extract article text, title, author, and publication date from any URL.
Proxy IP + Fingerprint Browser: The Ultimate Anti-Ban Combo
Fingerprints solve who you are. Proxies solve where you are. Both must work together to bypass modern anti-bot systems.
Self-Hosted Fingerprint Browser Cluster: Open-Source Multilogin Alternative
One Docker command to deploy a fingerprint browser cluster — an open-source, self-hosted Multilogin alternative.
Human Behavior Simulation: Bypass Behavioral Detection with humanize=True
One flag to enable human-like mouse, keyboard, and scroll patterns that bypass behavioral detection.
Fingerprint Management & Session Persistence: Make Your Scraper Look Like a Returning Visitor
Fixed fingerprint seeds, persistent cookies/sessions, and incognito bypass for maximum trust scores.
CloakBrowser Hands-On: Bypass Cloudflare Turnstile in 30 Seconds
Install, run, and bypass Cloudflare Turnstile with CloakBrowser in 3 lines of code.
Fingerprint Browser 101: Why Your Scraper Keeps Getting Blocked by Bot Detection
Understand browser fingerprinting and anti-detection, and see why C++-level patching is fundamentally different.
Issue Playbook: Crawler logs are cut by day in Proxy Routing and
Single-problem, engineering-grade Scrapy tutorial.
Issue Playbook: Missing the Scrapy entry in Wikipedia in m in Proxy
Single-problem, engineering-grade Scrapy tutorial.
Issue Playbook: S3FilesStore can use a lot of memory in Proxy Routing
Single-problem, engineering-grade Scrapy tutorial.
Issue Playbook: Scrapy needs to have sane defaults that do in Proxy
Single-problem, engineering-grade Scrapy tutorial.
Issue Playbook: Translating the docs in Proxy Routing and Health Scoring
Single-problem, engineering-grade Scrapy tutorial.