Designing Reconnect-Safe Live Captioning Systems with State Separation

Building real-time features like live captions in healthtech sessions presents unique challenges, especially when dealing with unreliable client connections. The core problem addressed is how to ensure accurate caption delivery and placement, even if a client temporarily disconnects and then reconnects. The proposed solution involves a strong separation of concerns to prevent data inconsistencies.

Separating Concerns for Reliability

The architecture advocates for three distinct systems, each with a specific responsibility:

• Real-time Channel: Responsible for moving fresh, short caption segments as they arrive. This channel prioritizes low latency and immediate delivery.
• Session Store: A dedicated system for holding the replayable caption and poll state for the active session. This acts as the authoritative source for reconstructing the UI upon client reconnection.
• Transcript Ledger: A separate, durable record assembled after the session ends. This ensures the long-term persistence and accuracy of the full transcript, distinct from the transient real-time experience.

Handling Reconnects and State Synchronization

Presence information (who is currently connected) is crucial for UI decisions like showing active users, but it's insufficient for recovery. When a client reconnects, it cannot assume it received all prior events. The proposed robust recovery mechanism involves:

1. The client identifies its last successfully processed sequence number.
2. It fetches the current authoritative session snapshot from the application's read boundary (the session store).
3. The client then resumes live consumption from the real-time channel, using the snapshot to close any data gaps and correctly place subsequent segments.

Data Structure for Captions

Each caption segment should carry essential metadata to ensure correct ordering and placement. A simple, vendor-neutral data contract is crucial:

from dataclasses import dataclass

@dataclass(frozen=True)
class CaptionSegment:
    session_id: str
    segment_id: str
    sequence: int
    start_ms: int
    text: str

    def validate(self) -> None:
        if self.sequence < 0 or self.start_ms < 0:
            raise ValueError("sequence and start_ms must be non-negative")
        if not self.text.strip():
            raise ValueError("caption text must not be empty")

The `sequence` provides a monotonic ordering for duplicate detection and ordering, while `start_ms` anchors the caption to the session timeline. Keeping segments short, ideally aligned with transcription stream boundaries and original media timestamps, enhances readability.