web-scraping

Web scraping for RAG ingestion. Covers Firecrawl API, Crawl4AI (open source), trafilatura (main-content extraction), BeautifulSoup + readability-lxml, Scrapy for large crawls, Playwright for JS-rendered pages, sitemap.xml discovery, robots.txt, polite crawling, anti-bot, and deduplication. USE WHEN: user mentions "web scraping", "crawl website", "Firecrawl…

Independent GitHub snapshot · Automated classification · Original documentation governs