10-Day Learn Scrapy Day 2: Core Engineering Lecture

Part 2: Selectors and Shell Debugging

This is Day 2/10 of "10-Day Learn Scrapy". Today solves one concrete problem only.

What Is Selectors and Shell Debugging?

Field Extraction and Detail-Page Join is a focused unit of scraping work that can be implemented and verified independently. Conclusion: you must deliver list-page plus detail-page combined schema by end of day.

Beginners Scrapy Tutorial

Constraints for this day:

single-module scope only
evidence must include commands, code, outputs, and validation
every failure needs one fix note

Today's repo documentation anchors:

scrapy/scrapy: key directories docs, extras, scrapy, sep
scrapy/scrapyd: key directories docs, integration_tests, scrapyd, tests
scrapy-plugins/scrapy-playwright: key directories docs, examples, scrapy_playwright, tests

Step 1 - Environment and Baseline Setup

cd ~/scrapy-labs/day01/bookslab
scrapy shell "https://example.com/catalogue/page-1.html"
scrapy crawl books -O output/day02.json

Step 2 - Build the Core Module

Core implementation snippet for today:

# parse list and follow detail
for card in response.css("article.product_pod"):
    detail_href = card.css("h3 a::attr(href)").get()
    item = {
        "title": card.css("h3 a::attr(title)").get(default="").strip(),
        "price_text": card.css("p.price_color::text").get(default=""),
    }
    yield response.follow(detail_href, callback=self.parse_detail, cb_kwargs={"item": item})

Step 3 - Run and Capture Outputs

Expected output check:

the crawl writes a structured output file;
critical fields are present and non-empty for sampled rows.

Step 4 - Validate and Fix Failures

Supporting code snippet for today's flow:

# parse detail page
def parse_detail(self, response, item):
    item["description"] = response.css("#product_description + p::text").get(default="").strip()
    item["upc"] = response.css("table.table-striped tr:nth-child(1) td::text").get(default="")
    yield item

Step 5 - Boundary and Acceptance

Pitfall 1: command success without data-quality checks.
Pitfall 2: manual visual inspection without scripts.
Pitfall 3: multi-variable changes in one experiment.

Acceptance table:

Check	Pass Criteria	Failure Signal	Fix Direction
Output size	>= 200 rows	far below threshold	inspect pagination/request path
Field quality	missing ratio <= 5%	many empty title/url	revisit selectors and cleaning
Validation script	pass	assert fail	debug failed rows and rerun
Rollback	recover in 10 min	irreversible changes	keep baseline config

Next Steps

Summarize today's knowledge coverage: core concepts, module implementation, validation and troubleshooting, production boundary
Record one failure and one fix action
Continue to the next Part with the same Step rhythm