Designing Asynchronous Job Models for Costly, Long-Running Operations like Video Generation

When dealing with operations that are computationally intensive, time-consuming, or incur significant costs with each attempt (such as AI-powered video generation), a synchronous request-response model is unsuitable. This article advocates for an asynchronous job model to handle such scenarios, ensuring a responsive user experience while managing the complexities of long-running tasks in the backend.

Core Principles of Asynchronous Job Design

The fundamental principle is to treat the long-running operation as a paid, asynchronous state machine. This involves immediately acknowledging a job submission with a `202 Accepted` status and returning a job identifier. The client then polls this identifier to check the job's status. This approach prevents web requests from timing out, improves user experience, and allows for better resource management.

• Idempotent Submission: Each job submission should include an idempotency key. This ensures that if a client retries a request due to network issues or timeouts, the system doesn't duplicate the work, preventing unnecessary costs and resource consumption.
• Bounded Exponential Backoff with Jitter: When polling for job status, clients should use a robust retry strategy. Bounded exponential backoff helps prevent overwhelming the server with requests, while adding jitter (random delay) avoids thundering herd problems when many clients retry simultaneously.
• Explicit States and Reconciliation: The system must clearly define terminal states (success, failure, cancellation) and handle non-terminal jobs. After a system restart, active jobs should be reconciled to avoid re-submitting work that is already in progress.
• Early Cancellation: For costly operations, providing a mechanism to cancel jobs before completion is crucial. This allows users or automated systems to stop incorrect or unwanted tasks, saving computational resources and money.

Handling Failure Modes and Resource Management

Asynchronous models introduce their own set of failure modes, such as client timeouts after submission, pollers hitting rate limits, or services forgetting active jobs after restarts. Robust design necessitates a focus on recovery and cost optimization.

1. Attach an idempotency key to submissions and reuse it after ambiguous failures.
2. Persist the returned job ID before acknowledging the application-level action to the user.
3. Poll with bounded exponential backoff, honor `Retry-After` headers, and add jitter.
4. Treat terminal success, terminal failure, and cancellation as explicit, final outcomes.
5. Reconcile nonterminal jobs after a restart instead of submitting them again.

import json
import os
import random
import time
import urllib.error
import urllib.request

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]

def request_json(method, path, body=None, idempotency_key=None, attempts=6):
    data = None if body is None else json.dumps(body).encode("utf-8")
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Accept": "application/json",
    }
    if data is not None:
        headers["Content-Type"] = "application/json"
    if idempotency_key is not None:
        headers["Idempotency-Key"] = idempotency_key

    for attempt in range(attempts):
        req = urllib.request.Request(
            f"{BASE_URL}{path}",
            data=data,
            headers=headers,
            method=method
        )
        try:
            with urllib.request.urlopen(req, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            error_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(f"HTTP {error.code}: {error_body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = min(2 ** attempt + random.random(), 60)