When collection breaks off midway, it never has to start over from the beginning

Background

The data to collect had piled up for more than five years: records kept weekly from the first week of 2019 through July 2024. It is far more than can be fetched at once, and if it stops midway, you first have to figure out where you left off. There was a deadline attached, so timing mattered.

What we saw on site

The problem was not the collection tool but the way requests were sent. If requests are fired in a burst, regardless of the speed a person can read, responses break off midway and that day’s collection ends up half-finished.

The problem we defined

We reframed the problem from “getting it fast” to “getting it all”. Rather than stopping at half, receiving the entire range — even if it takes longer — is what is useful. Data with missing segments has to be fetched again anyway.

Checking the target’s terms of service and robots exclusion standard is a precondition for that judgment. Unless it is settled first whether the data may be collected, everything designed afterwards is meaningless.

What we built

• Request interval and retry control — sends requests spaced out rather than in bursts, so five years’ worth comes through to the end without gaps in the middle
• Weekly segmented collection — instead of treating a span of over five years as one chunk, it is divided by week. If something fails, only that week needs to be fetched again
• Resume from the interruption point — it remembers where it stopped. A long collection can only finish if it never restarts from the beginning
• Remaining time display — progress and estimated time left on the screen. Run a multi-hour job with no indication and a person stops it midway

Result

Collected the entire range, week 1 of 2019 through week 30 of 2024, with no gaps. The same structure is now being used for other collection jobs.