India investment project collector
We collected every project from the government portal listing investment projects across India and compiled the scattered details into a single table with 41 fields. The tool was delivered as an executable for staff to run themselves.
Projects collected
Before
Projects looked up one at a time
After
19,889 projects · 41 fields
To protect client information, these screens recreate the same features with mock data.
Background
Information on investment projects in India is published on India Investment Grid, a government portal. The problem is that each project sits on its own page. Seeing how much investment is flowing into which sectors in which states meant opening the pages one by one.
What we found
The research needed the overall distribution, not individual projects. The portal is designed for looking up one project at a time, so viewing projects by sector or stage meant first copying everything out by hand. Given the number of projects, that work could not realistically begin.
The problem
Aligning the fields was harder than collecting them. Each project filled in different fields, project costs mixed currencies and units, and the same values appeared under different names. These differences had to be resolved before the data could fit in one table.
What we built
- Full project collection — Every project across 41 sectors was collected
- Normalization into 41 fields — Sector, sub-sector, project stage, total project cost, state, implementation model, project type, expected completion date, and other scattered values are placed in consistent columns
- Standalone executable — Delivered so staff can run it without a Python environment
Safeguards
At this volume, some requests will always fail. What matters is not whether they fail but whether the failures can be found later.
- Failed URLs are logged separately — If a request drops or gets no response, the URL is written to a separate file and collection continues
- A script to retry only that list — There is no need to rerun everything from the start. The second pass covers only what failed
- Retried results are stored with dates — There is a record of what was filled in and when
- 12 requests at a time, each with a timeout — One unresponsive request does not stop the whole run
Results
19,889 projects were organized into 41 fields. Of these, 19,087 have a confirmed total project cost.
By sector, roads and highways had the most projects at 4,180, followed by water, sanitation, and waste at 2,264, real estate at 1,480, health at 1,271, education at 954, and railways at 920. Projects fall into five stages, including under development, under implementation, and completed.
Related work is listed under Web data collection.