Customer story
Crawl efficiency up 340% for a national insurance platform, across 40,000 URLs.
A national insurance comparison platform carried 40,000 indexable URLs and spent most of its crawl budget on pages that earned nothing. Webzenia re-engineered crawl allocation and built a programmatic local layer so the platform's scale worked for it instead of against it.
The challenge
A Mumbai-based national insurance comparison platform carried 40,000 indexable URLs and wasted most of its crawl budget. Log analysis in Search Console showed Google spending 65% of its crawl on paginated archive pages that held no ranking value. The pages that mattered were visited too rarely to keep their rankings fresh.
At 40,000 URLs, crawl economics decide what ranks. Every crawl spent on an archive page was a crawl not spent on a commercial comparison page. The scale meant to be an advantage had become a tax.
The approach
Webzenia started with the logs. Botify and Screaming Frog mapped exactly where Googlebot was spending its time, confirming the drain into paginated archives.
The team restructured robots.txt and the internal-linking graph to steer crawl toward revenue pages and away from low-value archives. Pagination was consolidated, and crawl signals were pointed at the comparison templates that earn leads.
A programmatic local layer came next. Webzenia built location-specific comparison pages at national scale, each with unique data and on-page optimisation, so the platform could compete across cities rather than nationally only.
GA4, Search Console, and Looker Studio tracked crawl efficiency against organic sessions and leads, so the team could prove that crawl gains were turning into pipeline.
The results
Organic sessions grew from 84,000 to 290,000 a month, a 3.4x lift, in 8 months. Crawl efficiency rose 340% as Googlebot stopped burning budget on archives and started crawling the pages that rank.
Organic leads climbed 67% year on year. The programmatic local pages turned the platform's scale into coverage, letting it compete city by city rather than nationally alone.
At this scale the gains compound. Better crawl coverage keeps a larger share of the 40,000 URLs fresh and ranking, so the lift builds on itself rather than fading.
The stack behind it
More customer stories
Your turn
Want results like these?
Tell us where you want to grow. We'll map the first 90 days and show you what's realistic for your market.
What happens next
- 1A 30-minute discovery callWe learn your goals and constraints
- 2A tailored growth planChannels, milestones, and a realistic timeline
- 3KickoffA senior team, not a junior handoff
Start the conversation