---

name: crawlers-and-wikidata-pipelines description: "Operate Agriculture_KnowledgeGraph crawler and Wikidata/weather data-acquisition pipelines safely." disable-model-invocation: true metadata: disco-role: operating license: GPL 3.0

Crawlers and Wikidata Pipelines

Use this sub-skill when the task is to inspect, run, repair, or validate the repository's crawler-driven data acquisition and intermediate CSV generation workflows for Hudong/Baike pages, Wikidata entity/relation alignment, weather relations, attributes, and tree source lists.

Load this when

• The user asks how to run a Scrapy spider, choose the correct working directory, or understand crawler outputs.
• The task involves hudong_pedia, entityRelation, wikidata_relation, new_node, weather_plant, city_weather, attributes, staticResult, or tree-list artifacts before Neo4j import.
• A pipeline failed because expected JSON/CSV files are missing, a Scrapy project cannot find settings, or CSV rows are malformed.
• You need a no-network validation check for relation/weather CSV headers and row invariants.

Route elsewhere when

• The next action is final Neo4j LOAD CSV, Cypher import order, graph constraints, graph query semantics, or tree/vector query APIs; route to the graph-query-and-data-management sub-skill.
• The task is PCNN relation-extraction dataset generation, deduplication, or TensorFlow training; route to the relation-extraction-pipeline sub-skill.
• The task is Django route operation, Mongo-backed tagging UI, THULAC NER behavior, or KNN/fastText entity labeling; use the corresponding sibling sub-skill.

Operating references

1. Start with crawler-workflows.md for Scrapy projects, DFS tree crawlers, legacy data-processing scripts, working directories, output names, and network constraints.
2. Use wikidata-processing.md for Wikidata relation conversion, new_node.csv, wikidata_relation*.csv, relation distribution analysis, and safe CSV validation.
3. Use weather-and-attribute-pipelines.md for city/weather, weather-to-plant, climate-list, and attribute-extraction artifacts.
4. Use troubleshooting.md for crawler-specific recovery steps, path-sensitive failures, service prerequisites, and malformed CSV symptoms.

Bundled safe script

• validate_relation_csvs.py checks required CSV headers, non-empty fields, duplicate/suspicious rows, and cross-file consistency for wikidata_relation.csv, wikidata_relation2.csv, new_node.csv, weather_plant.csv, and city_weather.csv. It performs no network or database calls.

Example safe checks:

python scripts/validate_relation_csvs.py --help
python scripts/validate_relation_csvs.py --self-test
python scripts/validate_relation_csvs.py --root path/to/csv-export-directory

Safety defaults

• Treat all live crawls as network-expensive and site-policy-sensitive. Do not start a broad crawl, disable robots compliance, or raise concurrency without explicit user approval.
• Prefer inspecting existing generated artifacts and running the bundled CSV validator before any live crawl or service-backed conversion.
• Run path-sensitive source scripts from their documented project directories. Many scripts write outputs relative to the current working directory and some execute work at module import time.
• Do not claim Wikidata/weather conversion is verified unless the required local files and, where applicable, Neo4j/THULAC dependencies were actually checked in the active environment.