We receive it differently on each platform, and the result comes out as one table
At the request of a global research firm, we collected product reviews from twelve locations — major retail platforms across North America and Latin America, and home appliance brands' own online stores. Each platform delivered data in its own way, so instead of taking on all of them with one collector, we received each one differently and merged it all into a table in the same format.
Review data secured
Before
Separately, in a different way per platform
After
Twelve locations as one table in a single format
Background
In market research, product reviews are raw material. But the places that hold reviews are mostly large retail platforms with the strictest access conditions. Even when a survey was fully planned, work repeatedly came to a stop at the stage of getting the data in hand.
What we saw in the field
Each platform delivered data in its own way. Some were sensitive to request intervals, some returned different responses depending on the access environment, and some served only the review section as a separate API. If you try to take on all of them with one collector, it is unstable everywhere.
The problem we defined
We decided to use a different entry approach per platform, but to tie the cleanup, merging, and output after that into one. A collection coming to a stop is a problem of an individual platform; inconsistent output formats are a problem of the whole.
What we built
- Platform-specific collection engines — built separately for the major retail platforms of the US and Brazil and home appliance brands’ own online stores, each matched to how that site delivers data
- Values do not drift when the layout changes — for some platforms, instead of reading the visible markup, we referenced the internal data the page already holds, directly via a Chrome extension
- Request distribution and 3-stage parallel collection — requests are not sent in a burst from one place; the work is split across paths. The design is left in documentation so it can be reproduced
- Single output — no matter the platform, everything lands in a table of the same format
What we guarded against
We handled in code the things that actually happen in collections running for days, and verified that they behave as intended with 26 verification cases.
- Resume after interruption — restarts from whichever is smaller: the saved progress point or the count actually accumulated. Finding missing data later costs more than receiving overlaps
- Deduplication on merge — merged by unique identifier. Data already received stays as it is
- Path switching — if a response does not come, or comes with a broken format, switch to another path and continue from that point
- Data that does not match the shape never piles up in the table — if a received value does not match the expected format, it is filtered out on the spot
- Automatic cap on parallelism — if the configured value is too high, it lowers itself. Not hammering is the way to carry a multi-day collection to the end
Result
Even with twelve platforms, the result comes out as one table in a single format. Columns do not change depending on where the data came from, so the received data is used directly as analysis material, without a step of reshaping it. When one more target is added, the only thing that grows is that platform’s entry approach; the cleanup, merging, and output afterward are untouched.
Other work done the same way is collected under web data collection.
Sources
Outsourced work contract