Workload Attestation for Bridging Cloud and Internal Identity Systems at Scale

Organizations often manage two distinct identity systems: one provided by the cloud vendor (e.g., AWS IAM) and an internal system (e.g., a private PKI for mTLS). Bridging these two securely, especially for workloads on managed compute where direct control over bootstrapping is limited, presents a significant system design challenge. Netflix addressed this by designing a robust workload attestation mechanism for Spark jobs on Amazon EMR.

The Challenge: Bridging Disparate Identity Systems

Netflix's internal service-to-service authentication relies on Metatron, a private PKI issuing short-lived X.509 certificates. Workloads authenticate via mutual TLS. Crucially, a workload must first *attest* its identity to the identity service to receive a certificate. On managed platforms like EMR, a Spark job initially only possesses an AWS execution role, lacking the internal identity required for accessing sensitive resources (e.g., encrypted data, ACL-controlled tables).

Architectural Solution: Trustworthy Workload Attestation

The core of Netflix's solution involves a 1:1 mapping between internal "Data Project" identities and dedicated AWS IAM roles. This mapping, maintained by a Data Project service, allows translating a cloud provider's statement ("this process is running as role R") into an internal statement ("this process is workload W"). To handle tens of thousands of Data Projects without hitting IAM role limits, they shard roles deterministically across a pool of dedicated AWS accounts.

Attestation Workflow Overview

1. Control Plane Claim (Step 1): The control plane, authorized to launch Spark jobs, resolves the Data Project's IAM role, builds a signed workload metadata payload (including application identity, mapped role, scope), and passes it as job configuration. Only this service can produce a valid signature.
2. Workload Proves Cloud Identity (Step 2): The Spark driver plugin, using its AWS execution role credentials, generates a pre-signed URL for `sts:GetCallerIdentity`. This URL is a transferable proof of possession, as only the role holder could have created it. The plugin sends both the pre-signed URL and the signed metadata to the Identity service.
3. Corroboration & Issuance (Step 3): The Identity service performs four actions: 1) fetches the pre-signed URL via AWS STS to get the AWS-reported role, 2) verifies the control plane's metadata signature, 3) corroborates that the AWS-reported role matches the role in the signed metadata, and 4) issues short-lived X.509 certificates for the workload's internal identity. The workload never vouches for itself.

Scaling for Spark's Fan-Out Problem

Spark jobs involve one driver and potentially thousands of short-lived executors. Independent attestation by each executor would lead to significant load amplification on AWS STS and the internal identity service, resembling a DDoS attack. Netflix chose for executors to inherit credentials from the driver. The driver attests once and securely distributes credentials to executors via Spark's authenticated and encrypted internal RPC. This shifts the trust boundary to the driver but prevents massive, bursty attestation requests.

Credential Lifecycle Management

Certificates are short-lived. Since managed compute lacks OS-level hooks for renewal, the Spark driver JVM runs a timer to re-attest on a fixed interval before expiry. Executors do not renew; if they outlive their credentials, they exit and are replaced, which is more cost-effective. A crucial lesson: attestation must be a repeatable operation, not just a bootstrap step, to prevent long-running jobs from failing due to expired credentials.