Resumable data collector
Weekly records going back to 2019, more than five years of data, had to be fetched in one go. Requests sent in bursts get cut off partway and leave the day's collection half done. This collector resumes from the point of interruption and fills the entire range.
Range collected
Before
Stopped partway
After
Full range, 2019–2024
To protect client information, these screens recreate the same features with mock data.
Background
The data had built up over more than five years: weekly records from the first week of 2019 to July 2024. That is far too much to fetch at once, and if a run stops partway, the first job is working out where it left off. The schedule was fixed, so timing mattered.
What we found
The problem was not the collection tool but how requests were sent. Requests sent in bursts, far faster than anyone reads, get cut off partway, and the day’s collection ends up half done.
The problem
We changed the goal from “fetch it fast” to “fetch all of it”. A complete range that takes longer is worth more than half a range that stops. Data with gaps has to be fetched again anyway.
Checking the site’s terms of use and robots exclusion rules comes before any of this. If it is not settled first that the data may be collected, the rest of the design does not matter.
What we built
- Request pacing and retries — Requests are spread out rather than sent in bursts, so five years of data comes through without gaps
- Week-by-week collection — More than five years is split into weeks instead of one large job. If a week fails, only that week is fetched again
- Resume from point of interruption — It remembers where it stopped. A long run only finishes if it never has to start over
- Time-remaining display — Progress and estimated time remaining stay on screen. A job that runs for hours with no indicator tends to get stopped halfway
Results
Every week from week 1 of 2019 to week 30 of 2024 was collected, with no gaps. The same structure is now used on other collection projects.