---

name: serialization-storage description: "Serialize, persist, stream, and route DocArray BaseDoc, DocList, DocVec, file store, S3, and FastAPI payload workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

DocArray serialization and storage

Use this sub-skill when a task needs to move DocArray objects across process, file, tabular, web, or object-store boundaries.

Route here for

• Choosing JSON, protobuf, bytes, base64, binary file, CSV, or DataFrame formats for BaseDoc, DocList, and DocVec.
• Preserving or deliberately changing DocVec tensor backend with tensor_type=... during deserialization.
• Saving and loading DocList or DocVec binaries, including streaming-safe single-document protocols.
• Using DocList.push(), DocList.pull(), push_stream(), and pull_stream() with file:// stores.
• Documenting S3 DocList store requirements and constraints before credentials or network access are available.
• Returning DocArray documents and JSON payloads from FastAPI with DocArrayResponse.
• Diagnosing missing proto, pandas, web, or aws extras and protocol mismatch failures.

Route elsewhere

• Schema design, predefined documents, nested document modeling, and tensor field choices before serialization: use sibling document-modeling.
• Vector indexes, nearest-neighbor search, document indexes, or vector database backends: use sibling vector-indexing.
• Live S3 operations that require credentials, buckets, endpoints, or network access: verify those separately in the future task before relying on them.

Read these bundled references

1. Start with Serialization reference for the protocol matrix, exact method names, DocVec tensor_type behavior, CSV/DataFrame caveats, and FastAPI response routing.
2. Use Storage reference for file:// and S3 DocList store behavior, streaming, list/delete operations, path requirements, and optional service constraints.
3. Use Troubleshooting when optional extras, protobuf unions, CSV tensor fields, file namespace paths, local_cache, S3 credentials, or FastAPI imports fail.

Safe bundled helper

Run roundtrip_smoke.py in any environment where DocArray is installed to exercise safe local round-trips without external services:

Run these commands from this sub-skill root (the directory containing this SKILL.md):

python scripts/roundtrip_smoke.py --help
python scripts/roundtrip_smoke.py
python scripts/roundtrip_smoke.py --skip-protobuf --skip-dataframe --skip-file-store

The verified minimum scope is CPU with DocArray base plus proto, pandas, and web extras. Optional S3/AWS services and optional tensor frameworks are documented as boundaries, not verified defaults.