Where the company started
AI is only as good as the data feeding it, and most data foundations fall over under real load. Pipelines that resume where they stopped, validate before they index, and alert before they fail. This is where Cognilium began in 2019, and every AI system we build still stands on it.
2019
the year this practice started the company
250TB
Spark pipeline delivered on contract
~20
major retailers read daily by our own product
Four pipelines, described the way the record allows
A price-aggregation pipeline
Our own product: daily extraction of price, stock, ratings and reviews from roughly twenty major retailers, millions of product URLs worked concurrently, queued into Elasticsearch for search with daily history archived alongside.
A 250TB Spark pipeline
Delivered on contract — a platform-verifiable fact, with no client named.
Search-index pipelines
Document store to JSONL to bulk loaders with schema validation, field-enforcement exclusion and deduplication; large object-store joins against a relational warehouse; rate-limit-aware batching against third-party data APIs.
Streaming and orchestration
Airflow, Glue, Kafka, Kinesis, Spark, Celery and Redis, in real use rather than on a capability list.
Six patterns that decide whether it survives a bad night
The interesting engineering is not the moving of data. It is what happens when a source misbehaves halfway through a run of millions of records.
Offset-resumable jobs
A crash resumes from the last processed record. Without this, every failure at hour nine costs you hours one through eight as well.
Progress tracking and distributed queues
State held where every worker can see it, so scaling out does not mean re-doing work.
Exponential backoff
A struggling source is given room rather than hammered, which is also how you stay welcome.
Concurrency semaphores
Parallelism with a ceiling, because the fastest pipeline is not the one that gets your addresses blocked.
Batch-size-triggered flushes
Writes that happen on a threshold rather than per record, which is most of the difference between a pipeline that keeps up and one that does not.
Validation before indexing
Schema checks and deduplication run first. A silently missing value is a fault to raise, never a zero to store.
Where it runs is a budget decision
Managed Kubernetes costs a lot, and it has its benefits.
For a business with room in the budget, managed orchestration is worth what it costs in operational calm. For a business without it, the same containers scale horizontally on dirt-cheap hardware and the pipeline does not care. We will tell you which one your volumes actually justify.
What sits on top
Web scraping
The collection tier feeding these pipelines, with six escalation levels and the cost of each.
See it →Retrieval
Once the data is clean and indexed, the questions people actually ask become answerable.
See it →Cloud and DevOps
Three clouds, each with infrastructure as code, and cost engineering treated as a feature.
See it →Bring the pipeline that keeps falling over
This is our AI engineering practice
It is real work and it is where our four products came from. But what Cognilium leads with is narrower: optimization apps that run in tandem with Microsoft Dynamics 365, computing the decisions the ERP records but does not derive — the optimal price, the optimal pick path, the optimal stock level. See the optimization apps · How we build inside the ERP.