Enhancing Security Log Pipelines with MITRE ATT&CK Context
This article explores how an Observability Pipeline can enrich security logs with MITRE ATT&CK tactics and techniques before they are routed to a Security Information and Event Management (SIEM) system. It highlights the architectural benefits of pre-enrichment, such as reducing SIEM load, improving detection accuracy, and standardizing security data processing at scale within a distributed logging infrastructure.
Read original on Datadog BlogIn modern distributed systems, security observability is paramount. As logs from various services and infrastructure components are generated, effectively processing, enriching, and routing them to security tools like SIEMs becomes a significant architectural challenge. This article discusses an approach to enrich security logs at the pipeline level with MITRE ATT&CK context, providing immediate value for threat detection and analysis.
The Challenge of Security Log Processing
Security logs often arrive in a raw, unstructured, or semi-structured format, lacking the immediate context required for effective threat hunting or automated detection. Sending all raw logs directly to a SIEM can lead to: 1) High Ingestion Costs: SIEMs typically charge based on data volume, making raw log ingestion expensive. 2) Alert Fatigue: A flood of uncontextualized alerts can overwhelm security teams. 3) Slow Detection: Manual correlation and enrichment within the SIEM delay incident response.
Architecting an Observability Pipeline for Security Enrichment
An Observability Pipeline serves as an intermediate layer between log sources and destinations, enabling transformations, filtering, and enrichment. For security logs, this pipeline can be designed to perform pre-processing steps, reducing the load on the SIEM and enhancing the quality of ingested data. The core idea is to apply MITRE ATT&CK context to logs *before* they reach the SIEM.
- Ingestion: Collect logs from various sources (e.g., cloud platforms, applications, network devices) into the pipeline.
- Parsing and Normalization: Structure raw logs into a consistent format.
- Enrichment: Apply MITRE ATT&CK tactics and techniques based on log attributes (e.g., event codes, process names, command-line arguments). This might involve lookup tables or rule-based engines.
- Filtering and Routing: Discard irrelevant logs or route specific log types to different destinations (e.g., security logs to SIEM, debug logs to object storage).
- Delivery: Forward enriched logs to the SIEM or other long-term storage solutions.
System Design Trade-offs
Implementing an observability pipeline introduces a new component, adding complexity. However, the trade-offs include significant cost savings (reduced SIEM ingestion), improved operational efficiency (faster detections), and a more robust security posture. Consider stateless vs. stateful processing within the pipeline for enrichment, and ensure high availability and scalability for the pipeline itself.