Event schema evolution: how to change message formats without breaking consumers
Ivan Sato
·29 views
we're constantly dealing with event schema evolution across our data pipelines. adding new fields is usually fine if consumers are resilient, but removing or renaming fields often breaks things downstream, even with nullable options. we use avro with a schema registry, which helps with backward and forward compatibility for structural changes. however, it doesn't solve semantic changes. if we redefine what a 'user_id' means, consumers will still interpret it the old way. we're trying to figure out the best versioning strategy. should we use major/minor versions in the topic name, or embed versioning in the schema itself? what are the practical implications of each approach for a system with dozens of consumers?
6 comments