data-sourcesSelect, validate, and use Earth2Studio gridded, forecast, observational, tabular, satellite, cloud-store, and local data sources without confusing source contracts or variable lexicons.
Independent GitHub snapshot · Automated classification · Original documentation governs
Use this skill when a workflow must choose or call an Earth2Studio data source, forecast source, tabular observation source, or local Xarray source. It teaches source selection, request-shape validation, lexicon translation, time windows, and safe offline checks. It does not run model inference, select a prognostic/ diagnostic model, or perform full remote downloads.
Choose the smallest contract that matches the data:
| Need | Interface | Call | Result |
|---|---|---|---|
| Gridded analysis/initial state | DataSource | (time, variable) | xarray.DataArray |
| Forecast product with explicit origin and lead | ForecastSource | (time, lead_time, variable) | xarray.DataArray |
| Sparse observations or metadata | DataFrameSource | (time, variable, fields=None) | pandas.DataFrame |
| Sparse forecast observations | ForecastFrameSource | (time, lead_time, variable, fields=None) | pandas.DataFrame |
A DataSource's array normally has time, variable, then spatial/product
coordinates. A ForecastSource adds lead_time; DataFrame sources expose
SCHEMA, and fields must be a subset. Do not pass a forecast source to an
API expecting a plain source, or vice versa.
For a source choice, record product family, time coverage/cadence, variable vocabulary, grid, output type, cache policy, optional dependencies, credentials, and network policy. The built-in catalogue is broad but not exhaustive; availability, licenses, historical range, and variable support vary. See api-reference.md for representative families and decision-level prerequisites.
datetime, list, or NumPy datetime array is
accepted. Time normalization converts timezone-aware values to naive UTC;
naive values are treated as UTC.timedelta, list, or NumPy timedelta array as
lead_time; preserve the distinction between initialization time and
valid time time + lead_time.lead_time. Do not assume every source supports every ID,
cadence, level, or lead.fields after checking
source.SCHEMA. A DataFrame can contain observation values, time,
variable, lat, lon, elev, station/type, quality, or product-specific
metadata; it is not an N-D grid.The source's synchronous __call__ is normal. If it supports fetch,
await source.fetch(...) has the same contract for an async caller or bounded
batch. Do not assume every source has an async implementation.
Earth2Studio IDs such as t2m, u10m, v10m, msl, z500, and tcwv are
standardized vocabulary entries, not promises that every source provides them.
Each source has a source-specific lexicon mapping an Earth2Studio ID to a
backend key, selector, or compound route. A lexicon may also attach a modifier
for unit, sign, decomposition, accumulation, or representation conversion.
Before a remote or local request:
z500, t850) as pressure levels, m suffixes
(u10m) as height above surface, and k-style native levels as
source/use-case-specific. Do not interpolate or compare custom levels as if
they were interchangeable.The mapping patterns and a missing-variable synthetic example are in lexicon-reference.md.
Use these families for routing, not as an exhaustive model/data list:
WB2ERA5 variants),
NCAR/ERA5, CDS, CMIP6, and similar products. Check the data extra, date
range, cache, and grid/level support. Large requests may download or cache
substantial data.data extra plus service-specific availability and may require
public-cloud access, STAC metadata, or credentials. Confirm the sensor
lexicon and spatial coordinates before requesting data.DataArrayFile, DataSetFile, DataArrayDirectory,
DataArrayPathList, InferenceOutputSource, Constant, Constant_FX,
Random, Random_FX, and RandomDataFrame. Prefer these for reproducible
smoke tests, air-gapped runs, and pre-staged datasets.Remote constructors may be cheap while construction; the first call can open stores, list objects, download bytes, decode products, and populate a cache. Keep credentials, network policy, cache location, and data license explicit; never put credentials in a lexicon, source constructor, command line, or skill.
Use fetch_data(source, time, variable, lead_time=..., device=..., interp_to=..., interp_method=..., legacy=...) to obtain inference-ready data. It detects a
lead_time parameter; a plain DataSource is called at time + lead and
assembled with a lead dimension. legacy=True returns (torch.Tensor, CoordSystem); legacy=False returns Xarray, rejects interp_to, and needs
CuPy for CUDA. Curvilinear interpolation requires interp_method="linear".
Use fetch_dataframe(source, time, variable, fields=..., lead_time=..., device=...)
for tabular sources; it attaches request metadata in df.attrs, returns pandas
on CPU, and needs CuPy/cuDF on CUDA. prep_data_array converts an existing
DataArray to (tensor, CoordSystem). Exact signatures are in
api-reference.md; formats are in
data-formats.md.
time and variable, plus
lat/lon or documented spatial coordinates when needed.DataArrayFile(path) or DataSetFile(path, array_name); use
DataArrayDirectory for the year/month convention and DataArrayPathList
for consistent multi-file collections (requiring time, variable,
lat, and lon).time and lead_time, use InferenceOutputSource after
filtering extras so one of those dimensions has length one.--help and tiny-fixture check; it
never contacts a remote store.Example shape (the exact spatial dimensions are source-dependent):
from collections import OrderedDict
from datetime import datetime
import numpy as np
from earth2studio.data import Constant, Constant_FX, fetch_data
# A local deterministic fixture; no network or model package is needed.
source = Constant(OrderedDict([("lat", np.array([0.])), ("lon", np.array([0.]))]))
da = source(datetime(2024, 1, 1), ["t2m"])
forecast = Constant_FX(OrderedDict([("lat", np.array([0.])), ("lon", np.array([0.]))]))
x, coords = fetch_data(forecast, np.array([np.datetime64("2024-01-01")]),
np.array(["t2m"]),
lead_time=np.array([np.timedelta64(0, "h")]))
The snippet is intentionally a shape pattern, not an inference workflow. For a fuller local file check, use scripts/local_source_smoke.py.
A time tolerance is a selection policy, not a resampling instruction. Sources
that support it accept one timedelta/np.timedelta64 for symmetric
[-tol, +tol], or a (lower, upper) tuple for asymmetric bounds. Bounds must
satisfy lower <= upper; negative lower and positive upper are normal. The
observation row timestamp must be checked against the intended interval, and
empty results are not proof that a nearby record exists.
TimeWindow wraps a plain DataSource (not a ForecastSource) and fetches fixed
offsets, appending configured suffixes such as _tm1, _t, _tp1. It requires
non-empty equal-length offsets/suffixes; group_by controls output order.
Underlying missing-time and missing-variable behavior is preserved. See
troubleshooting.md before widening a window or
changing a source.
Classify failures before retrying: missing lexicon ID, invalid schema/field, unsupported time/lead, empty tolerance match, missing optional package, credential/network failure, or malformed local dimensions. Fix the class-specific cause; do not guess a backend name or silently widen time bounds.
This sub-skill intentionally omits model inference, full remote acquisition, credential setup, exhaustive source/model matrices, and backend-specific network operations. It does not guarantee that a named family is installed or that a public endpoint is reachable. For source construction, use the repository's separate datasource-development workflow; for model execution, route to the model/inference skill. Candidate native checks include the random, constant, local-Xarray, TimeWindow, and lexicon tests; they are evidence for validation planning, not a replacement for a request-specific source check.