Antenor Challenge
In marketplaces and classifieds, pricing decisions are only as good as your view of the competition, and that view goes stale in hours. Building it manually is impossible: competitor sites list millions of items, change their markup without notice, and actively resist automated access. A crawler that works today breaks next week, and a broken crawler that fails silently is worse than none, because the business keeps making decisions on data that stopped updating.
MercadoLibre, OLX, Comparar.com and Dridco all needed the same thing: a continuous, reliable picture of competitor listings and prices, at the scale of entire marketplaces.
Solution
Antenor built distributed crawling systems designed for sustained operation, not one-off scraping. The architecture treated breakage as a certainty rather than an exception: per-site extraction logic monitored for structural changes, scheduling that balanced freshness against target-site load, and pipelines that detected when a source started returning garbage instead of silently ingesting it.
Raw extraction is only half the problem. The crawlers fed normalization and matching pipelines that aligned the same product across different sites, formats and naming conventions, turning millions of scraped pages into feeds that pricing and category teams could act on directly.
Stack
- Distributed crawling infrastructure with scheduling and politeness controls
- Per-site extraction with structural change detection and breakage alerts
- Product matching and deduplication across sources
- Normalized price and listing feeds delivered to analytics teams
Results
- Price intelligence in production for four leaders of Latin American e-commerce and classifieds: MercadoLibre, OLX, Comparar.com and Dridco
- Powered Comparar.com, a public price-comparison site covering Argentina and Mexico
- Continuous competitor coverage replacing manual spot checks
