Designing a Zero-Allocation Worker Pool for High-Throughput Sensor Ingestion

The Challenge: High-Frequency Sensor Data Ingestion

The core problem addressed is ingesting 10,000 optical frames per second from a biosensor without dropping samples or incurring significant latency. This requires a robust system capable of handling high throughput and precise timing, specifically in an embedded, offline context where resource management and predictability are paramount. Two primary constraints guided the design: preventing the receive loop from stalling due to slow downstream processing and eliminating garbage collection (GC) pauses on the hot path to avoid silent data drops.

Architectural Principles for Real-time Data Streams

• Decoupling Producer and Consumer: The ingestion (gRPC receive loop) must be entirely independent of the processing (DSP handoff) to ensure that backpressure from a slow consumer does not propagate upstream to the sensor, potentially stalling data acquisition.
• Zero-Allocation Hot Path: To avoid unpredictable stop-the-world GC pauses, which can lead to missed samples at 10kHz, the critical path for processing each frame must be allocation-free. This ensures consistent performance and predictability.
• Bounded Resource Usage: In an offline, embedded system, unbounded resource consumption (e.g., goroutines or memory) under load is unacceptable. The system must degrade gracefully by applying backpressure rather than crashing due to resource exhaustion.

The Worker Pool Design

The solution implements a fixed-size worker pool responsible for processing incoming `OpticalFrame` data. Data is pushed into the pool via a gRPC client-streaming call and then passed to a DSP forwarder. A key component is a buffered channel (`jobQueue`) that acts as a buffer between the gRPC handler and the workers, allowing the ingestion to proceed without blocking while providing headroom for downstream processing. The `sync.Pool` mechanism is crucial for achieving zero allocations per frame on the hot path.

type FramePool struct {
    jobQueue chan *pb.OpticalFrame
    wg sync.WaitGroup
    forwarder *dsp.Forwarder
}

var frameSyncPool = sync.Pool{
    New: func() any { return make([]byte, 0, 1024) },
}

func NewFramePool(workers, queueSize int, forwarder *dsp.Forwarder) *FramePool {
    p := &FramePool{
        jobQueue: make(chan *pb.OpticalFrame, queueSize),
        forwarder: forwarder,
    }
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go p.worker(i)
    }
    return p
}

func (p *FramePool) Enqueue(frame *pb.OpticalFrame) {
    p.jobQueue <- frame // One channel send, blocks if queue is full
}

Zero-Allocation Workers with sync.Pool

Inside the worker goroutines, `sync.Pool` is utilized to recycle `[]byte` scratch space. Instead of allocating a new buffer for each frame and relying on the garbage collector, workers acquire a buffer from `frameSyncPool`, use it for processing, and then return it. This drastically reduces allocations on the hot path, effectively mitigating GC pauses and ensuring stable, low-latency processing at high data rates.

Why Fixed-Size Workers?

• Resource Bounding: Unbounded goroutines, while simpler to implement initially, can lead to uncontrolled memory usage and potential Out-Of-Memory (OOM) errors under sustained load, especially if downstream processing bottlenecks. A fixed pool guarantees a predictable memory footprint.
• Contention Control: Downstream components, like the `dsp.Forwarder.Push` which holds a mutex for per-sensor accumulators, can become contention points. Capping the number of concurrent workers limits lock contention, ensuring more predictable performance under load.