Designing for Evolutionary Data Compatibility with Schemaboi
This article discusses challenges in achieving data interoperability in local-first software and proposes a novel approach to schema evolution using an experimental Rust implementation called Schemaboi. It emphasizes the importance of durable, self-describing, and statically typed data formats to ensure forward, backward, and "sideways" compatibility for long-term data independence from applications.
Read original on InfoQ ArchitectureThe core problem addressed is the limitations of current data management approaches: traditional local software confines data to a single device, while cloud services often lock user data into centralized platforms. The vision presented is a local-first operating system where applications can evolve independently, yet share and interpret a common data layer, giving users true data ownership. This necessitates a highly robust and evolvable data format.
Key Principles for Durable Schema Formats
- Self-describing: The schema must be embedded directly within the data file (e.g., file header), allowing any tool to inspect and interpret the data without external schema files.
- Statically Typed: Ensures long-term data durability and integrity, unlike weakly typed formats like JSON.
- Compatibility over Time: Must support forward, backward, and "sideways" compatibility.
- Preservation of Unknown Data: Unrecognized fields in newer data versions must be preserved by older applications rather than discarded.
- Efficient Network Synchronization: Optimized for streaming and network transfer, crucial for local-first and distributed contexts.
Schemaboi's Approach to Schema Embedding
Schemaboi stores the schema in the initial bytes of every data file. When a file is opened, this embedded schema is merged with the application's internal schema. This contrasts with systems like Protocol Buffers or Cap'n Proto, which rely on pre-shared schema definition files. This self-describing nature is a fundamental architectural decision for achieving application independence.
Understanding Data Compatibility Types
Beyond the standard notions of backward and forward compatibility, the article introduces sideways compatibility. This is a critical concept for distributed and decentralized systems, allowing different developers to add fields to data structures independently without requiring central coordination. It relies on globally unique field identifiers (instead of incremental tags) to prevent collisions during parallel schema evolution. This design choice is fundamental for fostering open, interoperable ecosystems.
Safe Schema Evolution Guidelines
- Safe Actions: Adding optional fields, introducing new enum variants.
- High-Risk Actions: Making a field mandatory, renaming fields, altering a field's type. These actions can break compatibility and should be avoided or handled with extreme care, preferring incremental enhancements over breaking changes.