Automated PII Detection Pipeline with AWS Step Functions and Macie

Organizations frequently handle large volumes of sensitive data, making automated PII detection crucial for compliance with regulations like GDPR and HIPAA. Manual classification is unscalable and error-prone. This solution presents an event-driven pipeline that automates the discovery and classification of PII in data ingested into Amazon S3, extending Amazon Macie's capabilities with custom identifiers for organization-specific data types.

Architecture Overview

The core of the system is an event-driven architecture utilizing several AWS services to create a robust and scalable PII detection workflow. When a new file lands in an S3 bucket, an EventBridge rule triggers a Step Functions state machine. This state machine orchestrates the entire PII scanning process, ensuring reliable execution and reporting.

• Amazon S3: Used for data storage across three buckets to maintain data lineage and isolate objects by processing state: raw, staged, and scanned.
• Amazon EventBridge: Acts as the event bus, detecting S3 object uploads and triggering the Step Functions workflow with advanced filtering capabilities.
• AWS Step Functions: Orchestrates the multi-step PII scan lifecycle, handling job creation, status polling, report generation, and object movement. It includes built-in retry logic for resilience.
• Amazon Macie: The primary service for detecting PII, supporting both built-in managed data identifiers and custom identifiers defined by the organization.
• AWS Lambda: Provides the serverless compute for various logic components, such as initiating Macie jobs, parsing findings, and generating reports.
• Amazon SNS: Delivers real-time notifications for high-severity findings, enabling immediate response from security teams.

Key Design Decisions

Several architectural choices enhance the pipeline's effectiveness and reliability:

• Dedicated Macie Jobs per Object: Instead of batching, a dedicated Macie classification job is created for each object. This enables real-time PII detection upon ingestion, critical for timely compliance.
• Three-Bucket Pattern: Data isolation is achieved through separate S3 buckets for raw (ingestion), staged (processing), and scanned (final, validated) data. This prevents mixing of unprocessed and validated data.
• Step Functions for Orchestration: Step Functions ensures reliable, sequential execution of the workflow steps with built-in error handling and wait states, crucial for managing asynchronous Macie job execution.
• Custom Data Identifiers: Extends Macie's out-of-the-box capabilities to detect domain-specific PII patterns unique to an organization, such as policy or member IDs.
• EventBridge Triggering: Using EventBridge over native S3 event notifications provides more robust event filtering, support for multiple targets, and cross-account routing, offering greater flexibility and control.