Architecting Offline-First Generative AI for Edge Deployments

Deploying generative AI to industrial edge environments presents unique architectural challenges, primarily due to unreliable or absent cloud connectivity. An offline-first approach is crucial, where AI inference occurs locally at the edge, while cloud services manage model customization, deployment orchestration, and continuous improvement. This distributed pattern necessitates careful coordination across AI/ML, IoT, and storage services, along with deliberate trade-offs to balance model capabilities, hardware limitations, and operational complexity.

Key Architectural Decisions for Offline-First AI

The fundamental decision in an offline-first AI architecture is how to customize a language model to be sufficiently small for edge hardware. This involves working with Small Language Models (SLMs) that fit within the compute and memory constraints (typically GPUs with 16 GB+ VRAM) of edge devices. The chosen customization strategy directly impacts architectural complexity, cost, and data pipeline requirements.

• Model Fine-tuning (FT): Lightweight adaptation of a pre-trained model for specialized tasks (e.g., Q&A format). Effective for teaching style and structure, but less so for injecting new domain knowledge.
• Continued Pre-training (CPT): Resource-intensive process extending base model training with domain-specific unlabeled data to embed new knowledge (e.g., technical manuals). Requires significant computational resources and larger datasets.
• Hybrid Approach (FT + RAG): Combines fine-tuning for task-specific behavior with Retrieval Augmented Generation (RAG) for up-to-date knowledge. RAG uses local document retrieval (e.g., ChromaDB with sentence-transformers) to provide accurate answers with citations, reducing hallucinations without retraining the core model. Crucially, the RAG pipeline can operate on CPU/SSD, dedicating GPU VRAM to the SLM.
• Hybrid Approach (CPT + RAG + FT): Optimizes the entire RAG pipeline by combining continued pre-training of the embedding model with fine-tuning of the language model for both accurate retrieval and high-quality responses.

Reference Architecture Overview

The reference architecture for an edge-deployed generative AI solution, exemplified by a manufacturing use case, adopts the Hybrid (FT + RAG) approach. This balances compact model size with dynamic knowledge retrieval. The architecture is logically divided into cloud-side preparation, deployment orchestration, and edge-side inference.

1. Cloud-side Preparation: Technical manuals and SOPs are stored in Amazon S3. Amazon Bedrock processes raw documentation into structured Q&A pairs for fine-tuning. Amazon SageMaker AI Pipelines orchestrates an FMOps pipeline to fine-tune an SLM (e.g., gpt-oss-20b) and stores the customized model artifact in S3.
2. Edge Deployment Orchestration: AWS IoT Greengrass facilitates secure deployment of the customized models and associated components to edge devices. Hardware selection (e.g., NVIDIA Jetson Xavier or g4dn.12xlarge) is critical, influencing model viability and serving strategies like Model Replication (for high concurrency, short queries) or Tensor Parallelism (for fewer users, longer interactions).
3. Edge-side Inference: A local Flask-based UI enables operators to query the system. The fine-tuned SLM runs on-premises using Ollama as the inference runtime, which supports quantized model formats (GGUF) for efficient, local inference within memory constraints. The RAG knowledge base (ChromaDB) also resides and operates on the edge device, ensuring offline data retrieval.