Optimizing API for Historical Data Retrieval: GitHub's Star History Redesign
This article details GitHub's engineering process to redesign their star history API, transforming it from an inefficient identity-based endpoint to a performant, aggregate-based solution. The key architectural shift involved moving from paginating through individual stargazer records to pre-filtering by time range before aggregation, significantly reducing database load and improving response times. This case study highlights critical lessons in API design, performance optimization, and understanding user needs versus existing data models.
Read original on GitHub EngineeringThe Problem with the Original Stargazer API
Initially, GitHub's API for retrieving repository star history, GET /repos/{owner}/{repo}/stargazers, was designed to answer "who starred this repository?" It returned a paginated list of individual users, each with an associated `starred_at` timestamp. This design had two major flaws:
- Performance Bottleneck: To reconstruct a star growth curve, clients had to make thousands of requests, one for each page of 100 stargazers. Critically, the database would count *all* current stars for pagination metadata on *every single request*, leading to extremely slow queries and timeouts for popular repositories.
- Privacy and Misuse: The API exposed individual user identities, which bad actors scraped for spam and phishing campaigns, creating a privacy risk.
Understanding User Needs and Design Evolution
After restricting access to the original endpoint due to privacy concerns, GitHub engaged with maintainers and tool builders. They discovered that users primarily needed the curve – an approximate signal of interest and growth over time – not individual identities. This crucial insight allowed them to decouple the historical trend data from user identity, enabling a more efficient, purpose-built API.
Design Lesson: Separate Data Concerns
Do not let an existing data model dictate how a new use case is served if that model introduces inefficiencies or privacy risks. Clearly identify the core data requirement for the new use case and design an API around it, even if it means transforming or aggregating existing data.
Architectural Approaches Considered and Implemented
Several approaches were considered to address the performance and privacy issues, each with its own trade-offs:
- Removing Identities, Keeping Events: This would return `starred_at` without user IDs. While it solved the privacy issue and allowed existing tools to continue working, it did not solve the performance problem, as clients would still need to paginate and aggregate thousands of records.
- Aggregating, then Paginating: Grouping stars into daily/weekly buckets and then paginating the smaller result set. The issue here was that the database would still perform the expensive aggregation over *all* stars before applying `LIMIT` and `OFFSET` for pagination. A query for `react/react` would still examine millions of rows on each request.
- Filtering by Time Before Grouping (Implemented Solution): The chosen design paginates by calendar week. The API translates page and `per_page` parameters into an indexed `created_at` date range. Stars are *first filtered* to this specific range, and *then* grouped into daily counts. This significantly reduces the data processed per request, as the database only examines stars within the requested time window.
/* Inefficient: Aggregates all stars before pagination */
SELECT DATE(created_at), COUNT(*)
FROM stars
WHERE repository_id = ?
GROUP BY DATE(created_at)
ORDER BY DATE(created_at)
LIMIT ? OFFSET ?;
/* Efficient: Filters by time range first */
SELECT DATE(created_at), COUNT(*)
FROM stars
WHERE repository_id = ? AND created_at BETWEEN ? AND ?
GROUP BY DATE(created_at)
ORDER BY DATE(created_at);This design allows one request to cover up to 30 weeks of history, drastically reducing the number of requests (e.g., from 2,511 to 24 for `react/react`) and the database load. While exceptionally dense weeks remain a scaling limitation, the overall system is significantly more efficient and safer.