Scaling Multi-Tenant Search with OpenSearch Service

Multi-tenancy in search applications, where each customer or entity has isolated data, introduces significant scaling challenges. Designs that work for hundreds of tenants often fail at hundreds of thousands or millions, leading to slow queries and ballooning costs. This problem is particularly acute in user-facing applications like fintech, where instant access to historical data is critical.

Common Pitfalls in Multi-Tenant Search Architectures

• Pure Logging Approach: Optimizing for write throughput with large, shared shards (e.g., 50GB) and daily index rollovers works for application logs but fails for tenant-specific queries. Each query fans out across all shards, leading to excessive scatter-gather coordination costs and high latency.
• Pure Search Approach: Giving every tenant its own index provides perfect isolation and fast queries but quickly overwhelms the cluster with metadata management at scale (tens of thousands of tenants). The master node becomes a bottleneck due to the sheer number of indices and shards.
• Naive Tiering Strategies: Using log-optimized tiering (e.g., UltraWarm for older data) for user-facing queries can introduce unacceptable latency (seconds instead of milliseconds) because it frequently pulls data from S3 on demand. This is fine for forensic investigations but catastrophic for interactive user experiences.

The Hybrid Multi-Tenant Architecture Solution

The key to scaling multi-tenant search is acknowledging that not all tenants are equal and routing requests to dedicated resource slices. This involves a hybrid strategy combining shared resources with custom routing for smaller tenants and dedicated resources for larger, 'whale' tenants.

Custom Document Routing for Small Tenants

For the vast majority of small tenants sharing indices, custom document routing is crucial. By default, OpenSearch distributes documents using a hash function, spreading a tenant's data across many shards. Custom routing allows specifying a tenant ID as the routing key, ensuring all data for a given tenant lands on a single (or a small subset of) shards. This dramatically reduces query fan-out, cutting networking costs and latency by up to 98% at scale. This becomes critical beyond tens of thousands of tenants. For larger tenants within a shared pool, `index.routing_partition_size` can spread data across a predefined subset of shards to balance load.

Dedicated Resources for Large Tenants

For 'whale' tenants with high query volumes and specific isolation needs, a dedicated slice of resources is essential. This can range from a dedicated index within the same domain to an entirely separate OpenSearch domain (a "cell architecture") or even a dedicated collection in OpenSearch Serverless. This isolation prevents large tenants from impacting the performance of smaller tenants.

Optimized Shard Sizing and Writable Warm Storage

Mixed workloads (write-heavy ingestion and read-heavy queries) require a balanced approach to shard sizing. Instead of 50GB for pure logs or 10-30GB for pure search, aiming for 30-40GB per shard typically balances ingestion throughput and query performance effectively. Furthermore, modern "writable warm storage" (like OpenSearch Optimized OI2 instances) addresses the mutability limitations of older UltraWarm tiers. It offers S3-backed storage with local caching and allows direct writes to warm data, resolving late-arriving data or corrections in seconds without costly migration round-trips to hot storage.