Bright Data MCP: Configure Tools and Collect Web Results
MCP connectivity, remote job completion, and useful business data are different outcomes. A dependable integration checks each one. This tutorial reads two expl
Engineering Blog
Category: Technical Guides
40 posts in this category.
Browser Harness: Query a Dashboard Using a Signed-in Chrome Session
A dashboard may already have a humanestablished login and filter selection. Starting a fresh browser just to read one table can introduce unnecessary authentica
Browser Use: Collect and Verify Information across Websites
A crosssite research task needs more than “browse and summarize.” Without explicit sources and failure rules, an agent can present remembered information as new
Browser4: Collect Dynamic Pages and Recover Failed Work
Dynamic extraction needs to distinguish the loaded page, rendered data, and any stored snapshot. Otherwise a browser can display new content while a query keeps
CloakBrowser Manager: Manage Independent Browser Profiles
Two test accounts should not accidentally share cookies, writable browser directories, or network settings. A profile manager gives these settings an explicit i
Extract Complex HTML Tables into Consistent CSV
A table that looks clear in a browser can become misaligned data. A merged region cell disappears from later HTML rows, a year is lost above a metric, or a dash
Crawlee Crash Recovery: Resume Work without Duplicate Writes
The difficult crash window occurs after data commits but before a request is acknowledged. A restart may replay that request. Appendonly output duplicates it; a
Crawlee Python: Dynamic Job Listings, Details, and SQLite Storage
A loaded page can still contain only an empty listing container. Repeatedly retrying that response does not replace understanding when JavaScript supplies the d
Crawlee TypeScript: Paginated Products, Detail Extraction, and JSON Export
A useful product crawler must connect three tasks: discover the intended products, extract the right fields, and explain whether anything was missed. Retrieving
Add Data Quality Gates to a Web Crawler
An HTTP 200 response and an existing export do not establish useful data. The response may be a login page, a price selector may collect shipping, or a category
Expose a Crawler as a Job API with Progress and Cancellation
A synchronous HTTP wrapper leaves callers waiting for a crawler and obscures its progress. A job API should return identity first, then support status, results,
Crawler Regression Tests: Detect Silent Extraction Failures
Silent failures are often worse than exceptions: a crawler finishes while collecting original rather than current prices or omitting a list item. This tutorial
Fingerprint Suite: Create and Inspect a Consistent Browser Environment
A browser setup can become inconsistent even when every individual option looks deliberate: HTTP reports one UserAgent, JavaScript another, or language headers
Firecrawl: Discover, Crawl, and Export a Help Center
A hosted scraping API can remove browser process management from your application. It does not remove the need to retain job identity, wait for execution, retri
FoxProfile: Manage Independent Browser Tasks through REST
A backend browser job needs more than page actions. It must know which profile it owns, whether another job is using it, and who stops the browser afterward. Th
GOST v3: Share a Proxy Gateway across Crawlers
Maintaining the same upstream settings across Python, Node, and browser clients makes switching and diagnosis harder. GOST can expose shared clientfacing listen
Monitor Web Changes without Treating Failures as Deletions
Missing from today's output does not necessarily mean deleted. Incomplete discovery, detail timeouts, and broken selectors can all reduce a batch. This tutorial
LLM Scraper: Consistent Product Data across Page Layouts
When products appear across many layouts, maintaining one CSS rule set per template can become expensive. LLM Scraper can extract a shared schema, but it still
Maxun: Record Pagination and Export Data without Code
A nocode recorder can turn a small directory into a repeatable extraction task. A correct preview, however, does not establish a correct saved run, and forty ex
Moli: Fetch Pages and Export Content from the Command Line
A short list of pages does not always justify a crawler project. Moli provides commandline page material that can feed document processing or another extraction
mubeng: Check a Proxy List and Use Rotating Proxies
A proxy list is not automatically a usable pool. Likewise, reaching a checker endpoint does not establish that a target collection task will succeed. This tutor
Reconcile Product Data across Websites
Two websites selling “classic black tea” may offer different products or packages. Even 500 g and two 250 g boxes are not necessarily interchangeable offers. Th
Multilingual Scraping: Decode Pages and Export Correct CSV
When Chinese text becomes unreadable, locate the damaged stage first: response decoding, a later conversion, or the application opening the exported file. Savin
nodriver: Download, Check, and Merge Reports
A “download started” message is not a completed report. Closing the browser immediately or reading an old file can create an incomplete yet plausible summary. T
Build an Offline Documentation Mirror with Working Links
A folder of page text is not necessarily an offline mirror. Images may remain online, chapter links may break, and unrelated index pages may overwrite one anoth
Playwright Dynamic Lists: Responses, Scrolling, and Cursors
An infinite list may retain only visible nodes. Reading the DOM once at the end can miss earlier data, while an unchanged count may indicate failure rather than
Playwright MCP: Query and Filter Websites with an AI Assistant
A browsercapable assistant can filter a directory, open details, and organize evidence. It still needs current page state instead of a guessed sequence of eleme
Collect Product Variants: SKUs, Prices, and Availability
A product's initial price often belongs to only its default SKU. Reading that state once can miss variants, mix stock levels, and confuse invalid combinations w
proxy-chain: Connect Tools to an Authenticated Upstream Proxy
Some tools accept a proxy address but cannot conveniently express separate upstream credentials. proxychain can create a local forwarding endpoint and authentic
proxy.py: Build a Custom Forwarding Gateway for Crawlers
A forwarding gateway can centralize a request rule that several collectors would otherwise implement independently. proxy.py provides Python plugins for this pu
Pydoll: Search Dynamic Pages and Collect Paginated Results
A click, an idle network, and an updated result list are different states. Fixed sleeps can waste time on fast pages and still read stale content on slow ones.
Build a Python Async Scraping Pipeline with Backpressure and Continuous Output
Creating thousands of tasks and keeping every response until gather() finishes combines two problems: downloads can outrun parsing, and completed results remain
Build a Web Knowledge Base for RAG: Search and Trace Sources
Downloaded Markdown is not yet a dependable retrieval source. Chunks need provenance, refreshed pages must replace older chunks, and temporary collection failur
Reader: Add Web Article Extraction to a Node.js Application
Adding web articles to search or summarization becomes awkward when every function invents its own loading and extraction behavior. One returns empty text, anot
Collect Regional Product Data without Mixing Markets
Changing a locale does not guarantee a currency change. A regional exit also does not establish shipping eligibility. Sites may use cookies, accounts, and their
ScrapeGraphAI: Extract Company Profiles with Natural Language
Company information appears in different places across different websites. ScrapeGraphAI can turn a naturallanguage request into a fetching and extraction pipel
Migrate from Scrapy to Crawlee Python Without Changing Your Data Contract
The risky part of a crawler migration is not replacing parse() with another callback. It is allowing a working replacement to change the data quietly: amounts b
Skyvern: Complete and Check a Multi-step Form Draft
Multistep forms connect actions through business rules. Totals update after inputs, uploads must finish before progression, and validation can leave a task on a
Spider for Rust: Build a Site and Resource Inventory
A site inventory needs more than addresses: response status, saved material, and the resources each page declares all help later migration and diagnosis. This t
webclaw: Deploy an Internal Web Crawling Service
When several internal tools need page material, a shared extraction service can avoid duplicating browser and parser setup in every caller. webclaw provides bot