web-scraping

【文档与办公】Web scraping for RAG ingestion. Covers Firecrawl API, Crawl4AI (open source), trafilatura (main-content extraction), BeautifulSoup + readability-lxml, Scrapy for large crawls, Playwright for JS-rendered pages, sitemap.xml discovery, robots.txt, polite crawling, anti-bot, and deduplication. USE WHEN: user mentions "web scraping", "crawl website", "Fi…

独立 GitHub 快照 · 自动分类 · 以原作者文档为准