Building a Decentralized Content Distribution and Plugin Sync System
This article details the architecture and implementation of a content distribution system designed to synchronize blog posts and plugin information across multiple disparate websites. It covers critical aspects like content transformation, canonical URL management for SEO, and event-driven automation for plugin updates, highlighting practical solutions for maintaining consistent content and code across a distributed web presence.
Read original on Dev.to #architectureThe article describes the evolution of a content distribution system for Intent Solutions, aiming to centralize content management and automate publishing across three public-facing websites and a plugin marketplace. The initial problem was content silos, requiring manual effort to share or update content across sites with different technologies (Hugo and Astro) and data structures.
Content Transformation and Synchronization
A key challenge was converting content between different static site generators: Hugo (TOML frontmatter, specific directory structure) and Astro (YAML frontmatter, different directory structure). The solution involved content transformation, including frontmatter conversion, path rewriting, and image reference updates, to enable a "write once, publish everywhere" model. This approach moves beyond simple embeds or iframes, creating actual content copies tailored to each platform.
SEO and Canonical URLs
When distributing the same content across multiple domains (e.g., a professional post on a corporate site and its original on a blog), managing SEO is crucial to avoid duplicate content penalties. The system implements canonical URL overrides, where each distributed post explicitly points back to its original source URL. This tells search engines which version is the authoritative one, preserving SEO credit for the source.
Event-Driven Plugin Marketplace Sync
For the plugin marketplace, the article highlights a shift from a cron-based daily synchronization to an event-driven model using GitHub's `repository_dispatch` events. This architectural change significantly improves the freshness of plugin data by triggering immediate updates in the marketplace whenever a source plugin repository receives a `push` to its main branch. This eliminates manual triggers and reduces latency from hours to minutes.
on: push: branches: [main] jobs: notify: runs-on: ubuntu-latest steps: - uses: peter-evans/repository-dispatch@v3 with: token: ${{ secrets.MARKETPLACE_SYNC_PAT }} repository: owner/marketplace-repo event-type: external-plugin-update client-payload: '{"repo": "${{ github.repository }}"}'