flowchart LR subgraph Ingestion IngestionGateway["Ingestion Gateway (API)"] --> KafkaIngestionQueue["Kafka (Ingestion Queue)"] end subgraph Core Processing KafkaIngestionQueue --> ObjectStorage["Object Storage (S3-compatible)"] KafkaIngestionQueue --> PreprocessingService["Pre-processing Service"] PreprocessingService --> KafkaPreprocessingQueue["Kafka (Pre-processing Queue)"] KafkaPreprocessingQueue --> AIInferenceService["AI Inference Service"] AIInferenceService --> ExtractedDataStorage["Extracted Data Storage (PostgreSQL)"] end subgraph Data & Output ExtractedDataStorage --> SchemaManagementService["Schema Management Service (API)"] SchemaManagementService -.-> ExtractedDataStorage ExtractedDataStorage --> HITLService["Human-in-the-Loop (HITL) Service"] HITLService <--> HITL_UI["HITL User Interface"] HITLService --> KafkaHITLQueue["Kafka (HITL Review Queue)"] KafkaHITLQueue --> OutputIntegrationService["Output Integration Service"] OutputIntegrationService --> KafkaOutputQueue["Kafka (Output Queue for Downstream)"] OutputIntegrationService --> DownstreamAPIs["Downstream Systems (APIs)"] end subgraph Monitoring & Orchestration MonitoringSystem["Monitoring System (Prometheus/Grafana)"] <--- KafkaIngestionQueue MonitoringSystem <--- KafkaPreprocessingQueue MonitoringSystem <--- KafkaHITLQueue MonitoringSystem <--- KafkaOutputQueue MonitoringSystem <--- PreprocessingService MonitoringSystem <--- AIInferenceService MonitoringSystem <--- HITLService MonitoringSystem <--- OutputIntegrationService MonitoringSystem <--- ExtractedDataStorage OrchestrationEngine["Orchestration Engine (Prefect/Airflow)"] --> PreprocessingService OrchestrationEngine --> AIInferenceService OrchestrationEngine --> HITLService OrchestrationEngine --> OutputIntegrationService end