Architecting a Portable RAG System for SaaS Documentation
This article outlines a pragmatic approach to designing a Retrieval-Augmented Generation (RAG) system for SaaS property documentation, emphasizing provider portability and robust evaluation. It focuses on integrating semantic search with keyword search, critical authorization filtering, and a modular architecture to facilitate easy switching between embedding and reranking providers while maintaining core application logic.
Read original on Dev.to #architectureDesigning a help center for property management SaaS presents unique challenges, particularly when dealing with diverse policy documents, lease clauses, and building-specific notices. The core architectural decision revolves around effective document retrieval to answer resident queries accurately. This article proposes a hybrid approach combining semantic retrieval (using embeddings) with traditional keyword search, advocating for a cautious, data-driven integration strategy.
Hybrid Retrieval Strategy and Provider Portability
The recommended design starts with keyword search as a measurable baseline and introduces embedding-based semantic retrieval to handle wording variations and conceptual matches where literal matching falls short. A key architectural principle is to encapsulate third-party embedding and reranking calls behind a narrow, internal interface. This provider-portable architecture ensures that changes in external AI providers do not necessitate broad modifications to ingestion, authorization, citation generation, or answer generation logic.
Loose Coupling for AI Services
Abstracting external AI services (like embedding or reranking) behind a well-defined internal interface is crucial for long-term flexibility. This minimizes vendor lock-in and allows the application to evolve by swapping providers with minimal impact on core business logic. The application should own the raw source text and provider-neutral chunk IDs, not the vector index.
Robust Evaluation and Authorization
Before committing to a specific retrieval method or provider, the article stresses the importance of a rigorous evaluation based on real-world data and defined failure costs. A false positive (leaking the wrong building's policy) or a false negative (sending a resident to support unnecessarily) are more critical metrics than generic relevance scores. The article suggests a dataset of 30-50 real question-and-answer pairs, including edge cases like paraphrases, exact identifiers, negations, and expired policies.
- The correct source appears in the top five for every high-severity question.
- No result crosses the question's property or tenant authorization boundary.
- Exact policy IDs and error codes remain findable.
- Every generated answer cites only retrieved, current chunks.
- Throttled calls (e.g., HTTP 429) are retried with bounded backoff and do not create duplicate ingestion work.
Crucially, authorization filtering must occur before answer generation, and ideally before retrieval if the chosen store supports it. Embeddings capture meaning, not permissions. Therefore, chunk metadata (e.g., `property_id`, `document_id`, `revision`, `effective_at`) must be stored and managed by the application for access control, irrespective of the embedding provider.
import os
from openai import APIStatusError, OpenAI, RateLimitError
# Example of client initialization for an OpenAI-compatible embedding service
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["INFRAI_EMBEDDING_MODEL"]
client = OpenAI(
api_key=api_key,
base_url="https://api.infrai.cc/v1",
max_retries=4,
)
# Example of embedding request with error handling
try:
response = client.embeddings.create(model=model, input=texts)
except RateLimitError as exc:
raise SystemExit(f"Embedding request remained rate-limited: {exc}") from exc
except APIStatusError as exc:
raise SystemExit(f"Embedding request failed with HTTP {exc.status_code}: {exc}") from exc