Robust Change Detection for Dynamic Web Content

The Flaws of Naive Hashing for Change Detection

A common initial approach to detect changes on a web page is to hash its entire HTML body. While simple, this method is highly prone to false positives in real-world scenarios due to dynamic elements like rotating ads, user counters, CSRF tokens, and relative timestamps. These elements cause the page's hash to change on almost every poll, rendering the detection system useless for identifying actual content updates.

Entity Extraction and Set-Based Diffing

A more robust approach involves scraping the page to extract meaningful entities (e.g., `Listing[]` objects with title, price, URL) and then performing a set-based diff. The system only alerts when a new entity, identified by a stable key, appears in the current set that was not present in the previous set. This eliminates false positives from non-essential dynamic content.

Defining Entity Identity

Crucially, defining a stable identity for each entity is challenging. For example, a listing URL might need normalization to remove tracking parameters that change per page load. Engineering and product teams must collaborate to decide on trade-offs, such as accepting duplicate alerts versus missing genuine updates, to establish the most appropriate identity logic.

Ensuring Data Integrity with Atomic Writes

When persisting the previous state (snapshot) of entities, atomic write operations are essential to prevent data corruption. A process crash during a write can leave a truncated or malformed file, leading to severe issues like treating every item as new on the next poll (spamming users). The recommended pattern is to write to a temporary file and then atomically rename it over the target file.

function saveSnapshot(url: string, listings: Listing[]): void {
  const file = snapshotPath(url);
  const tmp = `${file}.tmp`;
  try {
    // Atomic write (temp + rename) so a crash can't corrupt the snapshot.
    fs.writeFileSync(tmp, JSON.stringify(listings, null, 2), 'utf8');
    fs.renameSync(tmp, file);
  } catch (err) {
    console.error(`[monitor] Failed to save snapshot for ${url}:`, err);
  }
}

Complementing atomic writes, robust read-side logic is necessary to handle potential file issues gracefully, including checks for file existence and validation of parsed JSON content to ensure it matches the expected data shape (e.g., an array of listings).