Observability in a 20-Year-Old MMO: EVE Online's Head-Based Sampling for "Fun"
EVE Online, a single-shard MMO with a 20-year-old architecture, details its approach to distributed tracing and observability. The article highlights their unique use of 100% head-based sampling, combined with dynamic exponential moving average (EMA) and deterministic sampling, to balance system fidelity, telemetry costs, and performance in a high-throughput, real-time game environment. This strategy addresses the distinct challenges of monitoring shared, transactional experiences in gaming versus traditional applications.
Read original on InfoQ ArchitectureUnique Observability Challenges in MMOs
Traditional software typically focuses on unique user experiences, whereas multiplayer online games like EVE Online must create shared, synchronous, and transactional experiences for potentially thousands of players in a single shard. This necessitates a different approach to observability, particularly when dealing with high-volume, ephemeral events such as in-game actions (e.g., bullet trajectories, ship skin changes). The EVE Online architecture, spanning 20 years, combines a legacy C++/SQL monolith with a newer Kubernetes-based microservices (or "domain services") ecosystem, presenting a complex monitoring landscape.
Head-Based Sampling with Dynamic & Deterministic Techniques
EVE Online notably employs 100% head-based sampling for distributed tracing, diverging from common tail-based sampling methods. This decision acknowledges tradeoffs but is managed by blending dynamic and deterministic sampling techniques. The core of their strategy involves dynamic exponential moving average (EMA) sampling, which allows them to adapt sampling rates based on real-time traffic and system load. Deterministic sampling is used for critical paths or events that must always be captured.
Head-Based vs. Tail-Based Sampling
Head-based sampling decides whether to sample a trace at its origin, before it propagates through the system. This is simpler but can miss traces that become interesting later. Tail-based sampling decides at the end of a trace, allowing for richer context, but requires collecting all trace data temporarily, which is more resource-intensive and complex for high-throughput systems like EVE Online.
Balancing Fidelity, Cost, and Performance
The primary goal of EVE Online's sampling strategy is to strike a balance between getting "good enough" fidelity for observing "fun" (player experiences) and controlling telemetry costs and performance impact. In a system processing 30,000 messages per second and simulating large-scale battles with thousands of players, collecting every event is infeasible and would significantly degrade performance. Their approach ensures that they can still gain insights into player interactions and system health without overwhelming their observability infrastructure.
The article highlights specific examples like players changing ship skins, which, despite seeming minor, can generate bursts of 50,000 messages per second, equivalent to the entire universe's typical load. Managing such bursts with intelligent sampling is crucial to avoid performance bottlenecks and manage data ingestion costs.
- Legacy Monolith: 1.7M lines C/C++, 2.6M lines SQL (MSSQL), 3.4M lines Stackless Python 2.7 (single logical processor).
- Modern Ecosystem (Quasar): Kubernetes, Go, gRPC, Protobuf, RabbitMQ, NATS.
- Throughput: Roughly 30,000 messages/second across both systems.
# Conceptual example of EMA-based dynamic sampling threshold adjustment
def adjust_sampling_rate_ema(current_rate, observed_load, ema_alpha=0.1):
target_rate = calculate_target_rate_based_on_load(observed_load) # e.g., higher load -> lower target sample rate
new_rate = ema_alpha * target_rate + (1 - ema_alpha) * current_rate
return max(0.01, min(1.0, new_rate)) # Ensure rate is between 1% and 100%
def calculate_target_rate_based_on_load(load):
if load > THRESHOLD_HIGH:
return 0.1 # Sample 10% during high load
elif load > THRESHOLD_MEDIUM:
return 0.5 # Sample 50% during medium load
else:
return 1.0 # Sample 100% during low load
# In EVE Online's case, 100% head-based means every trace starts,
# and sampling decisions are made at the origin based on dynamic factors.