Every price, every day
Checking competitors by hand means your prices always lag the market. This pipeline reads price, stock, ratings and reviews from roughly twenty major retailers on a daily cycle — millions of product URLs, queued, validated, indexed and served through an API.
It is our own product, so we can show you all of it.
~20
major retailers read on a daily cycle
4
fields per product: price, stock, ratings, reviews
0
records indexed before validation passes
Four fields, per product, per retailer
A price on its own is half the picture. What a competitor charges matters differently when the item is out of stock, and reviews are where product problems surface before returns do.
Price
What each retailer is asking today, per product, per variant.
Stock
Availability at the moment of collection, which is what makes a price actionable.
Ratings
The score a product carries where it is sold.
Reviews
The written record buyers leave, collected as text you can search.
Collect, queue, validate, index, serve
The interesting engineering is not the collecting. It is what happens when a source misbehaves halfway through a run of millions of records.
Collect
Millions of product URLs are worked through every day across roughly twenty major retailers, with the collection tier chosen per target rather than per project.
Queue
Everything lands in a distributed queue with progress tracked in Redis, so the work is resumable: a crashed job restarts from its last processed record, not from the beginning.
Validate
Schema validation, field-enforcement rules and deduplication run before anything is stored. A silently missing price is a fault to raise, never a zero to save.
Index
Batch ingestion into Elasticsearch so the catalogue is searchable in the shape buyers actually query it, with daily price history archived to S3 alongside.
Serve
An API over the top, so the feed is something your systems consume rather than a spreadsheet somebody downloads.
Where this runs is a budget decision, not a religion
Managed Kubernetes costs a lot, and it has its benefits.
For a business with room in the budget, managed orchestration is worth what it costs in operational calm. For a business without that room, the same containers scale horizontally on dirt-cheap bare metal and the pipeline does not care. We will tell you which one your volumes actually justify, and the answer is not always the expensive one.
Built with
The same discipline runs underneath our other pipelines, including a 250TB Spark pipeline delivered on contract.
Bring your catalogue and your competitors
This is our AI engineering practice
It is real work and it is where our four products came from. But what Cognilium leads with is narrower: optimization apps that run in tandem with Microsoft Dynamics 365, computing the decisions the ERP records but does not derive — the optimal price, the optimal pick path, the optimal stock level. See the optimization apps · How we build inside the ERP.