freehire: Architecture of an Open-Source Job Search Engine with Run-Once Workers and Meilisearch

Architectural Overview: Run-Once Workers for Scalable Background Tasks

freehire's most distinguishing architectural decision is its use of run-once workers instead of traditional long-lived job queue daemons for background tasks. This design minimizes the operational overhead of managing background runtimes. The system relies on PostgreSQL for coordination, utilizing transactional outboxes to ensure data consistency and reliability. When a worker completes a task, it writes its result and queues follow-up actions within the same database transaction. This atomic operation prevents data loss or inconsistencies if a worker crashes, as the entire operation is rolled back or committed together.

Crawler Design: Independent and Resilient Ingestion

Each ATS provider (e.g., Greenhouse, Lever) has its own configuration file, and the `cmd/ingest` worker processes one such file per run. This isolation means that slow-crawling platforms do not impede faster ones, enhancing the overall resilience and throughput of the ingestion pipeline. The system also implements a deduplication strategy using `jobs.UNIQUE (source, external_id)` in Postgres, allowing re-crawls to simply update a `last_seen` timestamp without re-indexing unchanged content.

Search Architecture: Meilisearch as the Primary Source

The search path is optimized for performance by storing the full job payload directly within Meilisearch documents. This eliminates the need for a separate database round trip to fetch job details after a search, significantly speeding up response times for search result pages. While descriptions are capped in the index for size management, the detail endpoint retrieves the full text from PostgreSQL. The system enforces deep pagination limits to prevent performance degradation.

// internal/search/searchdrain — search index document embeds the full job view
type JobDocument struct {
 jobview.Job // the same JSON shape the API returns
 // ...index-specific fields
}

Optimized Index Rebuilds and Deduplication

Instead of pushing individual changes to Meilisearch from many processes, the outbox pattern consolidates numerous small updates into fewer, larger pushes. This is crucial because Meilisearch re-merges its inverted index across the *entire* live index on every push. By reducing the frequency and increasing the batch size of these pushes, freehire mitigates disk I/O saturation and maintains search index performance, which was a significant issue in earlier designs with ~169 independent per-board processes.

AI/LLM Integration: In-Process and Secure

The in-app assistant runs directly in the same process as the HTTP server. This design choice simplifies deployment and enhances security by avoiding external agent runtimes, shell access, or separate credential management. Tools called by the assistant receive only the session owner's user ID and interact with the same Go services as regular HTTP handlers, limiting potential attack vectors like prompt injection by restricting outbound channels and disallowing actions like sending emails directly.