Global product review collector
For a global research firm, we collected product reviews from twelve sources across major North and Latin American retail platforms and home appliance brands' online stores. Each platform provides data differently, so each source is collected in its own way and merged into a table with one common format.
Review data acquisition
Before
Collected separately, platform by platform
After
Twelve sources in one table with a common format
To protect client information, these screens recreate the same features with mock data.
Background
In market research, product reviews are primary data. Most reviews, however, sit on large retail platforms with the strictest access conditions. Research plans were ready, yet projects repeatedly stalled at the data acquisition stage.
What we found
Each platform provided data differently. Some were sensitive to request frequency, some responded differently depending on the connection environment, and some served reviews through a separate API. A single collector built for all of them is unreliable on every one.
The problem
We decided to use a different entry method for each platform while handling cleansing, merging, and output in one pipeline. A stalled collection is a problem with one platform. Inconsistent output formats are a problem for the whole project.
What we built
- Platform-specific collection engines — Major retail platforms in the US and Brazil and home appliance brands’ online stores were each integrated according to how the site provides its data
- Values stay accurate when layouts change — For some platforms, a Chrome extension reads the internal data the page already holds instead of the visible layout
- Request distribution and three-stage parallel collection — Requests are spread across multiple routes rather than sent from one place. The design is documented so it can be reproduced
- Single output — Every platform produces a table in the same format
Safeguards
Issues that come up during collection runs lasting several days were handled in code and confirmed through 26 verification tests.
- Resume after interruption — Collection restarts from the lower of the saved checkpoint and the number of records actually stored. Collecting some records twice costs less than discovering missing data later
- Deduplication on merge — Records are merged by unique identifier. Data already collected is kept as is
- Route switching — If a response does not arrive or arrives malformed, collection switches to another route and continues from that point
- Malformed data stays out of the table — Values that do not match the expected format are filtered out immediately
- Automatic concurrency limits — If the configured level is too high, the system lowers it. Steady pacing is what carries a multi-day collection through to completion
Results
Twelve platforms produce one table in one format. Columns do not change with the source, so the data goes straight into analysis without a reformatting step. Adding a source means adding only that platform’s entry method. Cleansing, merging, and output stay unchanged.
Related work is listed under Web data collection.