spider

Crawl and scraping systems architecture — distributed crawler topology, URL frontier, politeness, and compliance. Architecture-only (no execution code). Don't use for single-page scraping (Navigator) or ETL pipelines (Stream).

Independent GitHub snapshot · Automated classification · Original documentation governs