Data Pruning and API Contract: The Silent Impact of Data Retention on System Behavior
This article highlights a critical system design issue where data retention policies, especially in systems relying on 'absence of data' for positive assertions, can silently alter an API's perceived contract. It details how pruning old, irrelevant data in a `gs_orgs` table, intended to improve database efficiency, inadvertently changed the meaning of an API's 'good standing' response, as the API could not differentiate between truly clear records and those pruned due to age. The core lesson is that data lifecycle management must be explicitly tied to API output contracts to prevent logical errors.
Read original on Dev.to #architectureThe Implicit API Contract of Data Absence
The article begins by describing an API for checking an organization's 'good standing' based on government revocation lists. Crucially, the underlying `gs_orgs` database table *only* stores organizations that are *not* in good standing. This means that a successful lookup resulting in 'no rows' implicitly signals 'good standing'. This design choice, while seemingly logical and efficient for storing only exceptions, creates an implicit API contract where absence of data carries significant semantic meaning.
Warning: Data Absence as a Signal
When an API's positive response (e.g., 'good standing') is derived from the *absence* of data in a database, any data retention or pruning strategy must explicitly account for this semantic implication. Failure to do so can lead to an API silently misrepresenting reality.
Data Pruning: Efficiency vs. Semantic Integrity
To reduce database size, a data retention policy was implemented to prune old revocation records (e.g., those from before 2020) that were no longer relevant for the product's specific use case (streamlined reinstatement paths). While this pruning was effective in reducing database size and keeping only 'actionable' data, it exposed a critical flaw. The API, when querying for an organization with a pre-2020 revocation, would now find 'no rows' and consequently report 'good standing' – a factually incorrect statement because the organization was still revoked, just not stored due to the retention policy.
keep_since = os.environ.get('GS_KEEP_REVOKED_SINCE', '2020-01-01')
def worth_keeping(r):
if r.get('ca_reg_no') or r.get('ca_registry_status'):
return True # CA-listed: AB 488 buyer at any age
if r.get('reinstatement_date'):
return False # back in good standing; "clear" is correct
return (r.get('revocation_date') or '') >= keep_since \
or (r.get('revocation_posting_date') or '') >= keep_sinceResolving the Discrepancy: Aligning API Output with Data Reality
The solution was not to unprune the data, which would negate the efficiency gains, but to adjust the API's output. The 'clear' message was updated to explicitly state the data's limitations: "No IRS auto-revocation posted since 2020..." This ensures that the API's assertion is truthful and consistent with the data actually being stored. This highlights the importance of data lineage and explicit data contracts in system design.
To prevent future drift between the retention policy (an environment variable) and the API's descriptive text, the author implemented a manual but effective solution: prominent comments in both the ingest code and the API route code. These comments explicitly warn developers about the dependency between the `GS_KEEP_REVOKED_SINCE` variable and the API's "clear" message, pointing to the interconnected components. While not a compile-time check, it serves as a crucial documentation and coordination mechanism in a distributed system context where multiple components' behaviors are intertwined.