10-Day Learn Scrapy Day 7: Core Engineering Lecture

Part 7: Proxy Middleware Integration

This is Day 7/10 of "10-Day Learn Scrapy". Today solves one concrete problem only.

What Is Proxy Middleware Integration?

Proxy Integration and Request-Path Validation is a focused unit of scraping work that can be implemented and verified independently. Conclusion: you must deliver a crawl path with verifiable proxy usage by end of day.

Beginners Scrapy Tutorial

Constraints for this day:

• single-module scope only
• evidence must include commands, code, outputs, and validation
• every failure needs one fix note

Today's repo documentation anchors:

• scrapy/scrapy: key directories docs, extras, scrapy, sep
• scrapy/scrapyd: key directories docs, integration_tests, scrapyd, tests
• scrapy-plugins/scrapy-playwright: key directories docs, examples, scrapy_playwright, tests

Step 1 - Environment and Baseline Setup

cd ~/scrapy-labs/day01/bookslab
export HTTP_PROXY="http://user:pass@proxy.example:8000"
scrapy crawl books -O output/day07.json

Step 2 - Build the Core Module

Core implementation snippet for today:

# middleware.py
class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = spider.settings.get("HTTP_PROXY")
        request.headers['X-Debug-Proxy'] = 'enabled'

Step 3 - Run and Capture Outputs

Expected output check:

• the crawl writes a structured output file;
• critical fields are present and non-empty for sampled rows.

Step 4 - Validate and Fix Failures

Supporting code snippet for today's flow:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "bookslab.middleware.ProxyMiddleware": 350,
}
HTTP_PROXY = 'http://user:pass@proxy.example:8000'

Step 5 - Boundary and Acceptance

• Pitfall 1: command success without data-quality checks.
• Pitfall 2: manual visual inspection without scripts.
• Pitfall 3: multi-variable changes in one experiment.

Acceptance table:

Next Steps

• Summarize today's knowledge coverage: core concepts, module implementation, validation and troubleshooting, production boundary
• Record one failure and one fix action
• Continue to the next Part with the same Step rhythm