---

name: design-serialization-schema locale: caveman-ultra source_locale: en source_commit: 82c77053 fence_basis_commit: 82c77053 translator: "Julius Brussee homage — caveman" translation_date: "2026-04-24" description: > Design serialization schemas using JSON Schema, Protocol Buffer definitions, or Apache Avro. Covers schema versioning, backwards compatibility, validation rules, and evolution strategies for long-lived data formats. Use when defining a new API contract or data interchange format, adding fields to an existing schema without breaking consumers, migrating between schema versions, choosing between schema systems, or documenting data validation rules for automated enforcement. license: MIT allowed-tools: Read Write Edit Bash Grep Glob metadata: author: Philipp Thoss version: "1.0" domain: data-serialization complexity: intermediate language: multi tags: json-schema, protobuf, avro, schema-evolution, versioning, compatibility

Design Serialization Schema

Versioned schemas → evolve w/o breaking consumers.

Use When

• New API contract / data format
• Add fields w/o break consumers
• Migrate schema versions
• Pick schema sys (JSON Schema, Protobuf, Avro)
• Doc valid. rules → auto-enforce

In

• Required: Data model (entities, types, constraints)
• Required: Compat reqs (consumers, old format lifetime)
• Optional: Existing schema → evolve
• Optional: Perf reqs (valid. speed, registry)
• Optional: Target format (JSON, binary, columnar)

Do

Step 1: Pick Schema Sys

Sys	Format	Strength	Best
JSON Schema	JSON	Broad support, flex valid.	REST, config
Protocol Buffers	Binary	Compact, fast, typed, evo built-in	gRPC, micro
Apache Avro	Binary/JSON	Schema in data, great evo	Kafka, pipelines
XML Schema (XSD)	XML	Deep typing, namespaces	Enterprise/SOAP
TypeBox/Zod	TypeScript	Type inference + runtime valid.	TS APIs

→ Schema sys picked → ecosystem + perf + evo reqs. If err: unsure → start JSON Schema (broadest tooling, layers on JSON APIs).

Step 2: Core Schema

JSON Schema ex:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/schemas/measurement/v1",
  "title": "Measurement",
  "description": "A sensor measurement reading",
  "type": "object",
  "required": ["sensor_id", "value", "unit", "timestamp"],
  "properties": {
    "sensor_id": {
      "type": "string",
      "pattern": "^[a-z]+-[0-9]+quot;,
      "description": "Unique sensor identifier (lowercase-digits format)"
    },
    "value": {
      "type": "number",
      "description": "Measured value"
    },
    "unit": {
      "type": "string",
      "enum": ["celsius", "fahrenheit", "kelvin", "percent", "ppm"],
      "description": "Unit of measurement"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp with timezone"
    },
    "metadata": {
      "type": "object",
      "additionalProperties": true,
      "description": "Optional key-value metadata"
    }
  },
  "additionalProperties": false
}

Protocol Buffers ex:

syntax = "proto3";
package sensors.v1;

import "google/protobuf/timestamp.proto";

// Measurement represents a single sensor reading.
message Measurement {
  string sensor_id = 1;         // Unique sensor identifier
  double value = 2;             // Measured value
  Unit unit = 3;                // Unit of measurement
  google.protobuf.Timestamp timestamp = 4;
  map<string, string> metadata = 5; // Optional key-value metadata
}

enum Unit {
  UNIT_UNSPECIFIED = 0;
  UNIT_CELSIUS = 1;
  UNIT_FAHRENHEIT = 2;
  UNIT_KELVIN = 3;
  UNIT_PERCENT = 4;
  UNIT_PPM = 5;
}

Apache Avro ex:

{
  "type": "record",
  "name": "Measurement",
  "namespace": "com.example.sensors",
  "doc": "A sensor measurement reading",
  "fields": [
    {"name": "sensor_id", "type": "string", "doc": "Unique sensor identifier"},
    {"name": "value", "type": "double", "doc": "Measured value"},
    {"name": "unit", "type": {"type": "enum", "name": "Unit", "symbols": ["CELSIUS", "FAHRENHEIT", "KELVIN", "PERCENT", "PPM"]}},
    {"name": "timestamp", "type": {"type": "long", "logicalType": "timestamp-millis"}},
    {"name": "metadata", "type": ["null", {"type": "map", "values": "string"}], "default": null}
  ]
}

→ Schema self-doc → descriptions + constraints + clear types. If err: data model unstable → mark draft, skip registry.

Step 3: Plan Evolution

Compat rules:

Change	Back Compat?	Fwd Compat?	Safe?
Add optional field	Yes	Yes	Yes
Add required field	No	Yes	No (breaks consumers)
Remove optional field	Yes	No	Careful (producers may still send)
Remove required field	Yes	No	Careful
Rename field	No	No	No (use alias + deprecate)
Change field type	No	No	No (add new, deprecate old)
Add enum value	Yes (if consumers ignore unknown)	No	Depends on impl
Remove enum value	No	Yes	No

Safe evo:

1. Only add optional fields w/ defaults
2. Never remove/rename → deprecate
3. Version schema in id (v1, v2)
4. Schema registry for binary (Confluent for Avro/Protobuf)

Protobuf evo rules:

// v1 — original
message Measurement {
  string sensor_id = 1;
  double value = 2;
  Unit unit = 3;
}

// v2 — safe evolution
message Measurement {
  string sensor_id = 1;
  double value = 2;
  Unit unit = 3;
  // NEW: added in v2 — old clients ignore this field
  google.protobuf.Timestamp timestamp = 4;
  // DEPRECATED: use sensor_id instead
  reserved 6;
  reserved "old_sensor_name";
}

JSON Schema versioning:

{
  "$id": "https://example.com/schemas/measurement/v2",
  "allOf": [
    {"$ref": "https://example.com/schemas/measurement/v1"},
    {
      "properties": {
        "location": {
          "type": "string",
          "description": "Added in v2: GPS coordinates"
        }
      }
    }
  ]
}

→ Evo plan documented: safe changes + version reqs. If err: break unavoidable → version (v1 → v2), parallel support during migration.

Step 4: Implement Valid.

# JSON Schema validation (Python)
from jsonschema import validate, ValidationError
import json

schema = json.load(open("measurement_v1.json"))

def validate_measurement(data: dict) -> list[str]:
    """Validate a measurement against the schema. Returns list of errors."""
    errors = []
    try:
        validate(instance=data, schema=schema)
    except ValidationError as e:
        errors.append(f"{e.json_path}: {e.message}")
    return errors

# Usage
errors = validate_measurement({"sensor_id": "s-01", "value": "not_a_number"})
# → ["$.value: 'not_a_number' is not of type 'number'"]

// TypeScript with Zod (runtime + compile-time)
import { z } from 'zod';

const MeasurementSchema = z.object({
  sensor_id: z.string().regex(/^[a-z]+-[0-9]+$/),
  value: z.number(),
  unit: z.enum(['celsius', 'fahrenheit', 'kelvin', 'percent', 'ppm']),
  timestamp: z.string().datetime(),
  metadata: z.record(z.string()).optional(),
});

type Measurement = z.infer<typeof MeasurementSchema>;

// Validation
const result = MeasurementSchema.safeParse(inputData);
if (!result.success) {
  console.error(result.error.issues);
}

→ Valid. on all incoming data at boundaries (API, ingestion). If err: log valid. errs w/ full payload (redact sensitive) for debug.

Step 5: Doc Schema

Schema doc page:

# Measurement Schema (v1)

## Overview
Represents a single sensor reading with metadata.

## Fields
| Field | Type | Required | Description | Constraints |
|-------|------|----------|-------------|-------------|
| sensor_id | string | Yes | Unique sensor ID | Pattern: `^[a-z]+-[0-9]+

  
    
    
    
    
    
    
    
    
    
    
    
  
  
    
 |
| value | number | Yes | Measured value | Any valid IEEE 754 double |
| unit | enum | Yes | Unit of measurement | One of: celsius, fahrenheit, kelvin, percent, ppm |
| timestamp | string | Yes | Reading time | ISO 8601 with timezone |
| metadata | object | No | Key-value pairs | String keys and values |

## Changelog
| Version | Date | Changes |
|---------|------|---------|
| v1 | 2025-03-01 | Initial schema |

## Compatibility
- **Backwards**: Consumers of v1 will continue to work with future versions
- **Policy**: Only additive, optional field changes between minor versions

→ Docs auto-gen or in sync w/ schema. If err: docs drift → CI check valid. docs vs schema source.

Check

• Schema sys matches use case (JSON Schema, Protobuf, Avro)
• All fields: types + desc + constraints
• Required vs optional explicit
• Evo strategy documented (safe changes, versioning)
• Valid. at boundaries
• Versioned + changelog
• Round-trip: serialize → deserialize → compare, no data loss

Traps

• Over-constrain early: Strict valid. on new schema → blocks iteration. Start permissive (additionalProperties: true), tighten later.
• No defaults: Add required field w/o default → breaks existing data. Always defaults for new fields.
• Null ignored: Many schemas sloppy on null/missing. Explicit nullable vs optional.
• Version in URL not payload: Long-lived data (storage, events) → embed ver in data, not just endpoint URL.
• Enum exhaustive: New enum val crashes consumers w/ exhaustive switches. Doc unknown → handle gracefully.

→

• serialize-data-formats — format pick + encode/decode
• implement-pharma-serialisation — pharma (regulatory schemas)
• write-validation-documentation — valid. docs for regulated schemas