Understanding AWS Lambda Durable Functions and Cold Start Impacts
This article presents a system design challenge involving an AWS Lambda durable function for a loan approval workflow. It highlights a critical issue caused by re-initializing a large ruleset outside of durable operations, leading to prolonged cold start times that ultimately prevent the function from completing its tasks within the configured invocation timeout. The case study serves as an excellent example of the hidden costs and complexities associated with serverless architectures and durable execution patterns.
Read original on Dev.to #systemdesignThe article describes a scenario where a lending platform uses an AWS Lambda durable function for its loan approval process. This workflow involves several steps, including application validation, credit report retrieval, and a human review loop with a 6-hour wait period, repeating for up to 5 days. The Lambda function has a 90-second invocation timeout, while the overall execution has a 7-day timeout.
The Hidden Problem: Ruleset Loading on Every Resume
A crucial architectural decision was to load a 40 MB compliance ruleset from S3 and parse it, taking approximately 50 seconds. This loading process was placed at the very top of the handler, *before* any durable operations. The engineer's rationale was to load it once since multiple steps needed it. However, this decision had an unforeseen consequence in the durable function's execution model.
Durable Functions and Replay Behavior
AWS Lambda's durable function (similar to Azure Durable Functions) uses a checkpoint-and-replay execution model. On every resume after a `wait` operation, the handler re-runs from the top. It skips completed durable operations by using their stored results. This means any code executed *outside* of these durable operations will re-execute on every single resume, including initialization logic like loading a large ruleset.
Impact and Failure Mechanism
Initially, the workflow functioned correctly. However, after several review cycles, applications stopped progressing. The critical clue was that invocation durations were consistently pinned at exactly 90 seconds – the function timeout. The 50-second ruleset load, combined with the growing time taken to replay the durable execution history, eventually exceeded the 90-second invocation limit before any new work could be processed. This caused the Lambda invocation to be killed, retried, and killed again, leading to a silent stall of applications.
System Design Lessons Learned
- Cold Start Amplification: In durable functions, cold start initialization logic that isn't part of a durable operation is re-executed on every resume, significantly amplifying its impact over long-running workflows.
- Invocation vs. Execution Timeout: Distinguish between a single Lambda invocation timeout and the overall durable execution timeout. Long-running workflows can be stalled by repeated invocation timeouts even if the overall execution is within limits.
- Idempotency and Side Effects: Durable operations should be idempotent. Code outside durable operations, especially costly initialization, must be carefully managed to avoid re-execution penalties. Consider caching, externalizing, or re-structuring initialization within the durable context where appropriate.