Technology

The stack we build on, and why each piece is there

Eleven layers, from the ERP integration surface up to the traces your on-call team reads, with the reasoning for every choice. Including the parts that are hard: what breaks in production, and what we build so that it does not take the rest down with it.

Talk to an Engineer AI in the ERP

layers, each with the choice explained

AI agents running in production across our own products

clouds in production, each with infrastructure as code

The stack, top to bottom

01ERP integration surface
02Orchestration
03Foundation models
04Retrieval and vector
05Voice
06Document intelligence
07Inference and serving
08Evaluation and MLOps
09Data plane
10Observability
11Cloud and delivery

Principles

Four things we will not trade away

These decide the architecture before any tool is chosen, which is why they come first on this page too.

If it does not run in production, it does not count

A demo proves a model can do something once. Production asks whether it still does it on the four-thousandth call, at 3am, when a provider is degraded and the input is malformed. Everything on this page is built for the second question.

The model is a configuration choice, not an architecture

Every system we build goes behind a thin provider abstraction, so swapping a model is a config change rather than a refactor. The same graph runs against a hosted frontier model, the same model on a cloud's own contract, or an open-weight model in an air-gapped cluster.

Cost is engineered down before deploy, not after the bill

Routing sends the easy majority of calls to a small model and reserves the large one for the hard remainder. Caching, deduplication and a scoring pass that decides what the expensive stage never has to see are all design-time decisions, not optimisations bolted on in month three.

Every answer carries the evidence for itself

Retrieval returns the passage, extraction returns the quote and the page, and a classification returns the reasoning that produced it. A system whose output cannot be checked is a system nobody is allowed to act on.

Layer 01 · ERP

What we know about the ERP, specifically

We do AI, and we do it inside Microsoft Dynamics 365, SAP and modern ERP. This is the layer that is scarce, so it goes first.

Two products that share a name

Dynamics 365 is two different things. Customer Engagement covers Sales, Customer Service and Field Service, built on Dataverse and the Power Platform with clean REST APIs. Finance & Operations covers Finance, Supply Chain Management and Commerce, and it is the real ERP core with a different data layer, different APIs and a different environment model. A team that asks for the wrong environment loses weeks before it touches any data.

The supported integration surface

On Finance & Operations: OData data entities for reading and writing master data, inventory and locations; custom services for logic that does not fit an entity; and Business Events for reacting to receipts and put-away as they happen rather than polling for them. Standard, documented, and still there after an update — which is the difference between an integration and a liability.

The authentication detail that stops most projects

Service-to-service through a Microsoft Entra app registration — and that registration has to be registered inside Finance & Operations and mapped to a Finance & Operations service role before a single API call will succeed. It is one screen, it is in the documentation, and not knowing it is the most common reason an AI team's first Dynamics sprint produces nothing.

Business Central, and what its API does not expose

Business Central online publishes a large set of standard API endpoints, and a handful of the things a real workflow needs are not among them — item references and price lists being the ones that come up first. The answer is a small companion extension covering exactly that gap, installed once, rather than a sync process that drifts.

The customer's own governed stack

Dynamics work runs where the customer's IT already governs it: Power Platform, Dataverse and Azure, with the tenant, the identity model and the data residency the customer already signed off. Nothing we add asks a security review to start over.

SAP and modern ERP

The same shape of problem and the same discipline: read through the platform's own supported surface, compute the decision outside the core, write back only what we created. The optimization patterns are not Microsoft-specific; the integration work is, and it is done against each platform's published contract.

The whole ERP lane is written up at AI in tandem with Dynamics 365, and the optimization apps it produces are at the Optimizers.

The stack

Ten more layers, and the reason for every tool

A stack list with no reasoning is a shopping list. Each entry below says what it is for and when we reach for something else instead.

Orchestration

Where the graph and the agent topology live

LangGraph: Default. Durable state, checkpointing to Postgres or Redis, and human-in-loop nodes — the only mainstream orchestrator with a recovery story you can defend to an operator.
CrewAI: Role-based collaborative agents, when persona separation is itself the product rather than an implementation detail.
AWS Bedrock AgentCore: Managed runtime for AWS-native customers, with IAM-scoped tool execution and guardrails already wired.
Google ADK: Vertex-native tool registration with built-in tracing, when the customer's ML budget already lives on GCP.
LangChain: Adapters and small utility chains. Not primary control flow — a framework that owns your control flow owns your debugging.

Foundation models

Routed per call, by cost, latency and capability

Anthropic Claude: Default for complex tool use, long-context reading, and work where a model knowing when to decline matters as much as its answer.
OpenAI GPT-4o and 4o-mini: Cost-efficient routing on high-volume paths, and multimodal vision where a frontier model is more than the job needs.
AWS Bedrock: Same models under the customer's existing AWS contract, when data residency, a BAA or private networking decide the architecture.
Google Gemini: Very long context for whole-corpus ingest in one pass, and a fast tier for high-volume summarisation.
Open weights on vLLM: Llama, Mistral and Qwen for air-gapped deployments and for a per-token cost floor nobody can raise on you.

Retrieval and vector

Hybrid search, reranking, graph

Qdrant: Most common in production here. Hybrid search with payload filtering at large scale, and cheap to self-host inside a customer's own network.
Pinecone: Managed serverless when there is no ops budget; namespace-per-tenant for multi-tenant products.
Elastic and OpenSearch: The keyword half of every hybrid setup, and usually the search backbone a customer already trusts.
Postgres pgvector: When the vectors belong next to the transactional data and a second store is not worth operating.
Cross-encoder rerankers: The stage that decides whether retrieval is any good. Managed rerankers where that is allowed, open re-rankers in-network where it is not.
Neo4j and property graphs: GraphRAG when the question is about relationships between things, which nearest-neighbour search answers badly. Temporal edges when the question is what was true on a date.

Voice

Real-time speech for telephony and live agents

Speech-to-speech models: Our default for low latency. A single model avoids the transcribe-think-speak stack, and that stack is where the awkward pause comes from.
Streaming speech-to-text: When we need the transcript as a first-class artefact — word-level timing, diarization, and analytics over what was actually said.
Neural text-to-speech: Branded voices, and multilingual output for deployments where the caller's language is not English.
Twilio and LiveKit: The carrier and WebRTC layer above the speech stack, with per-call quality telemetry and a fallback when the line is worse than useless.

Document intelligence

Extraction from PDFs, scans and forms

Cloud document processors: Forms, invoices and tables, from all three major clouds — chosen by which cloud the customer is already governed on, then paired with a model pass for the free text.
Layout-aware chunking: Turning messy PDF, DOCX and HTML into clean elements before anything embeds them. Most bad retrieval is bad chunking wearing a costume.
Multimodal parsing: For scanned documents, reading the page as an image rather than as extracted text. On one engagement that single change took extraction from roughly 12% to roughly 94% on the same documents.
Deterministic parsers: Spreadsheets and structured files are parsed with code, not a model. A rulebook that changes depending on which model read it is not a rulebook.

Inference and serving

Where open-weight models actually run

NVIDIA Triton: Multi-model serving so embeddings, rerankers and small models share a node instead of each holding their own.
vLLM: Paged attention throughput for open-weight serving, and the default engine behind Triton.
Managed endpoints: SageMaker or Vertex when the customer is all-in on one cloud and has no appetite for running an inference server.
Burst inference: For image, audio and infrequent open-weight calls, so nobody pays for a GPU that is idle most of the week.

Evaluation and MLOps

How we know it works before it ships

Versioned golden datasets: Hand-curated examples per task, versioned like code and reviewable in a pull request. Without one, 'it seems better' is the whole quality process.
Retrieval evaluation harness: Faithfulness, answer relevance, context precision and context recall, measured per change rather than argued about.
Model-as-judge with rubrics: Explicit rubrics, judge-versus-human agreement tracked, and more than one judge model — because a single judge shares its own blind spots with the thing it is judging.
Regression gates: A drop below the agreed bar blocks the deploy. The gate is agreed before the work starts, which is the only time it can be agreed honestly.
Experiment tracking and registry: Promoted runs become the canonical production pointer, so what is running is always answerable from a system rather than from memory.

Data plane

The foundations AI runs on

PostgreSQL: Default transactional store, with time-series extensions for event and metric data and pgvector when vectors belong here.
ClickHouse: Sub-second analytical queries over very large trace and event tables — the backbone of the dashboards that tell you what a system actually did.
Customer warehouses and lakehouses: Features and evaluation data land in the warehouse the customer already owns. We do not duplicate a company's data to make our own life easier.
dbt: Transformations as version-controlled SQL with tests — the contract between the data team and the ML team, written down.
Kafka and change data capture: The event backbone for streaming ingest and agent message buses, and low-latency capture from the transactional store so a retrieval corpus does not go stale.

Observability

Traces, evaluations, cost, prompts

OpenTelemetry GenAI conventions: Vendor-neutral spans across model calls, retrieval and tool calls, exported into whatever the customer's on-call team already watches. Their SRE should see a model span next to a service span, in their tool.
Prompt-level tracing: Hosted where the customer is happy with that, self-hosted where residency or an air gap decides otherwise. Same data either way.
Per-tenant spend metering: Token cost broken out by tenant, route and model from the first staging deploy, with budget alerts wired to wherever the team already gets paged.
PII redaction on logs: Prompts and responses are logged; that means they are a data-protection surface, and they get redacted before they land.
Trace replay: Any production trace can be re-run against a new prompt or a new model to see the diff before the change ships, rather than after.

Cloud and delivery

Where it runs, and how it gets there

All three major clouds: AWS, GCP and Azure each carry Cognilium systems in production, each with its own infrastructure as code. We build where the customer already is.
Customer-network deployment: A Terraform module and a Helm chart targeting the customer's own Kubernetes — the same artefact, a different variable file, so the review is about their environment rather than about us.
Air-gapped variants: Open-weight models served locally with local embeddings and no outbound model calls, for environments where an outbound call is the thing that fails the review.
Separated environments and gated deploys: Staging and production as separate stacks, with promotion to production gated by an explicit confirmation rather than by a merge.

Topology

Most agent failures are coordination bugs, not model bugs

Which is why the shape of the system is decided before the framework is, and decided from the task rather than from a tutorial.

Single agent with a bounded tool budget

A ReAct loop with a hard cap on tool calls. Right for a well-defined task with a handful of tools, and wrong the moment the task fans out — it loses state, exhausts context, and has no recovery story when a call fails halfway.

Supervisor and workers

The default for anything that decomposes. A router reads intent, checks what the caller is allowed to see, and dispatches to a specialist that owns one domain and its tools. Adding a capability means adding a worker, not editing a prompt everyone shares.

Plan, execute, critic

For work where the plan is worth reviewing before it runs. The critic is a separate role with a separate prompt, because a model asked to check its own output grades generously.

Parallel committee with a merge

Independent workers on the same input, then a merge step that resolves them, with early exit on consensus. Where they disagree is the useful signal — that is the case a person should see, and the disagreement is the reason why.

Per-node budgets

Every node carries a token cap, a time cap and a retry cap. Without them one adversarial input can drive an unbounded loop, and the first you know is the bill. Exceeding the budget fails closed to a handoff, never to a retry storm.

The choice is made from the task shape

Not from a default and not from whatever the framework's tutorial used. Most production failures in agent systems are coordination bugs, not model bugs, and the topology is where coordination is decided.

Before it ships

Six things that happen before any traffic reaches it

Including the one most teams skip: whether the system will act on instructions it finds inside a document it retrieved.

Retrieval before fine-tuning, nearly always

Fine-tuning answers a question most systems are not asking. The failures that actually happen are about freshness, citability, and the long tail of documents the model never saw — and none of those are fixed by adjusting weights. We start from a keyword baseline, add dense retrieval, add a rerank stage, and keep each step only if it measurably beat the one before.

Chunking chosen for the document, not for the default

Semantic, recursive or layout-aware, decided by how the source is actually structured. A fixed window across every document type is the single most common cause of retrieval that looks fine in a demo and fails on the real corpus.

Embeddings chosen by a bake-off on your corpus

Not by a leaderboard. Candidate embedding models are run against the customer's own material and their own questions, and the winner is whichever one wins there.

A red-team suite that has to pass before traffic

Jailbreaks, direct prompt injection, indirect injection arriving through retrieved content, and scope creep. The indirect case is the one most teams miss: a system that will act on instructions found inside a retrieved document has a security problem, not a prompt problem, and it is re-run on every release branch.

Cost projected at ten times the volume

Before launch, against the worst-case usage pattern rather than the average one. If the projection overruns the budget, the routing mix and the prompts are rewritten then — not after the first month's invoice makes the decision for us.

Shadow traffic, then a staged ramp

Run silently against real traffic first and compare. Then five percent, then a quarter, then all of it, with the primary metric checked for significance at each step before the next one opens. And a runbook — the top alerts, the debug paths, the fallback, who to call — handed to the customer's team in the first week, not at the end.

Reliability

What we build so that a bad hour is not a bad day

Every pattern here was learned the same way: something broke, and the fix turned out to generalise. This is the part of a stack that nobody demos.

Two-tier retry, and one honest failure state

A transient error retries inside the worker with exponential backoff. If that is exhausted the message goes back to the queue and is delivered again, to a different worker, with the same budget. Only when BOTH tiers are exhausted is the job marked permanently failed. One tier alone either gives up on a blip or retries a genuinely broken thing forever.

Circuit breakers with a half-open state

Closed, open, half-open. After a threshold of failures the breaker opens and calls fail fast instead of piling into a provider that is already struggling. After a recovery interval one probe call is allowed through, and its result decides whether the breaker closes or opens again. Without the half-open state you either flap or stay dark.

Proportional back-pressure beats a binary breaker

For rate-limited external systems we coordinate a counter across every worker, so the whole fleet shares one budget rather than each replica guessing. That gives continuous back-pressure: when the far side slows, fewer requests complete per window and the system throttles itself; when it recovers, throughput returns on its own. No threshold to misfire, no cooldown to tune.

Check the counter before you increment it

The obvious rate limiter increments first and then checks the limit — and every rejected retry adds a phantom count, so the counter runs ahead of reality and delays workers into the next window too. Reading the counter first and incrementing only on an allowed call keeps it exact. Both steps run inside one atomic script so two workers cannot interleave. It is a small bug with a fleet-wide blast radius.

Isolate the failure, then put a ceiling on it

Independent work runs in independent workers with independent error handling, so one degraded dependency cannot take the others with it. Above that sits a hard deadline: a task that hangs is killed and its slot returned to the pool. Isolation stops the spread; the deadline stops one slow dependency quietly consuming all the capacity.

A cache must never be able to break the system

Every cache read and write is wrapped so that a cache outage degrades to a slower correct answer rather than an error. Keys carry every parameter that changes the result, and the cache is consulted before a database connection is opened rather than after — which is where most of the saving actually comes from.

A worked pipeline

Eight stages, and stage three is the one people skip

Document extraction, as we actually build it. Finding the evidence before extracting the value is what makes the output checkable rather than merely plausible.

Parse

Raw text out of PDF, DOCX and spreadsheets with format-appropriate parsers. Scanned pages take the multimodal path instead.

Classify

Identify the document type against a taxonomy the customer defines, and return a confidence with it.

Find the evidence

Locate the passages that bear on each field BEFORE extracting anything. This ordering is what makes the output checkable.

Extract

Structured extraction of the fields that type calls for, against the evidence rather than against the whole document.

Validate

Cross-field rules — the arithmetic and the relationships that have to hold. A value that passes extraction can still be impossible.

Score

Confidence per field, not per document. One uncertain field should not devalue thirty good ones, and a human should be sent to the one.

Orchestrate

The stage that owns retries, partial failure and ordering, so a wobble in stage four does not lose stages one to three.

Write to the graph

Entities and relationships persisted, with each extracted value still carrying the quote and the page it came from. Reconciliation across documents happens here, where the entities already exist.

Written up at length on document intelligence, and running in production inside Paralegent AI.

Delivery

How a project actually ships

Six steps, and the metric that decides whether it worked is agreed in step one — because a success metric agreed afterwards is not one.

Diagnose

One workflow, the data behind it, and the measurement that will say whether this worked. Agreed in writing before anything is built, because a success metric agreed afterwards is not one.

Architect

The integration surface, the model routing, the evaluation set and the failure modes. This is where the ERP surface work happens, and where most of the risk is actually removed.

Build the first production slice

A pilot on one workflow, typically four to six weeks, with a weekly readout. No throwaway code: the pilot is the first slice of the real system.

Prove it against your data

The golden set is built from the customer's own material and the gates are run against it. This is where an accuracy number becomes real, and it is the only place one can.

Deploy where you already are

Into the customer's cloud and identity model, as infrastructure as code, with documentation, dashboards and runbooks. Handover is part of the build, not a phase after it.

Watch it, then tune it

Traces, spend and evaluation scores from day one, with a re-evaluation path for when production traffic drifts away from the set it was tested on.

Said before you ask

These are design targets, not a service commitment

They go into the project spec as gates the build has to clear. They are not an SLA, and we will not sell you one.

Single-model inference: p99 under 80ms, on a served open-weight model with cache reuse
End-to-end retrieval: p99 under 800ms, from question to first streamed token
Multi-agent committee: p99 under 2.5s for a small worker set plus a supervisor merge
Voice first token: under 600ms, which is where a caller stops talking over the system

On compliance, the same precision: our architectures are aligned with SOC 2, ISO 27001, GDPR, HIPAA and PCI-DSS practices, and we do not hold those certifications. What is real is the engineering underneath — least privilege, encryption in transit and at rest, role-based access, audit logging on every interaction, and redaction before a prompt is ever written to a log.

From the engineering blog

The decisions behind this stack, written up in full

Each of these is the long version of a choice described above — the trade-offs, the numbers we could defend, and the point at which the approach stops working.

Engineering FAQ

What engineering teams ask on the first call

Every answer is in the page source rather than behind a click, because the systems that summarise this page never click anything.

What does Cognilium actually know about Dynamics 365 that a general AI team does not?

The parts that decide whether a project starts. That Dynamics 365 is two products — Customer Engagement on Dataverse, and Finance & Operations, which is the real ERP core with a different data layer, different APIs and a different environment model. That the supported surface on F&O is OData data entities, custom services and Business Events, not screen automation. And that a Microsoft Entra app registration has to be registered inside Finance & Operations and mapped to a service role before a single API call will succeed — one screen, in the documentation, and the most common reason a first Dynamics sprint produces nothing.

Do you work with SAP as well?

We build AI for SAP and modern ERP platforms using the same discipline: read through the platform's own supported surface, compute the decision outside the core, and write back only what we created. The optimization patterns are not Microsoft-specific. The integration work is platform-specific and is done against each platform's published contract.

What is the default orchestration framework?

LangGraph, because it is the mainstream orchestrator with durable state, checkpointing to Postgres or Redis, and human-in-loop nodes — so a long-running task can be paused, inspected and resumed by a person. CrewAI when role separation is itself the product. AWS Bedrock AgentCore for AWS-native customers who want a managed runtime with IAM-scoped tool execution. Google ADK for Vertex-native deployments. LangChain is used for adapters and small chains, deliberately not for primary control flow.

Why Qdrant by default rather than a managed vector database?

Hybrid search and payload filtering at large scale, and it is cheap to self-host inside a customer's own network — which matters because a good number of enterprise engagements require that data never leaves the perimeter. Pinecone when there is no ops budget. Postgres pgvector at smaller scale, when running a second store is not worth the operational cost. The choice is made per engagement, against the customer's constraints, not once for everyone.

How do you know a system works before it ships?

A versioned golden dataset built from the customer's own material, held like code and reviewable in a pull request, plus an evaluation harness measuring faithfulness, answer relevance, context precision and context recall. Model-as-judge with explicit rubrics, judge-versus-human agreement tracked, and more than one judge model, because a single judge shares its blind spots with the thing it is judging. Regression gates block a deploy when the score drops below the bar — and the bar is agreed before the work starts, which is the only time it can be agreed honestly.

What keeps a system up when a provider degrades?

Two-tier retry, so a transient error retries in the worker and then again from the queue, and only a job that exhausts both is marked permanently failed. Circuit breakers with a half-open state, so recovery is probed rather than guessed. Distributed rate limiting that gives continuous back-pressure instead of a binary open-or-closed, coordinated so a whole fleet shares one budget. Failure isolation per worker with a hard deadline above it. And caches wrapped so that a cache outage degrades to a slower correct answer rather than an error.

How do you deploy into our environment?

A Terraform module and a Helm chart targeting your own Kubernetes on AWS, GCP or Azure — the same artefact with a different variable file, so your review is about your environment rather than about us. For regulated settings there is an air-gapped variant running open-weight models locally with local embeddings and no outbound model calls. Staging and production are separate stacks, and promotion to production is gated by an explicit confirmation rather than by a merge.

What is your compliance posture, precisely?

Precisely is the right word, so: architectures are ALIGNED WITH SOC 2, ISO 27001, GDPR, HIPAA and PCI-DSS practices, and Cognilium does not hold those certifications. What is real is the engineering — secure by design, least privilege, encryption in transit and at rest, role-based access, audit logging on every interaction, PII redaction on logged prompts and responses, and configurable audit retention. Where a BAA or a DPA is required, we build on a model provider that offers one under the customer's own contract.

How do you control what a system costs to run?

By designing it down before deploy rather than after the first bill. Routing sends the easy majority of calls to a small model and reserves the large one for the hard remainder. Prompt caching, semantic caching and cache reuse are on the launch checklist. Token spend is metered per tenant, per route and per model from the first staging deploy, with budget alerts wired to wherever the team already gets paged. On one contract-review pipeline, a scoring-and-routing stage cut model calls by 75% against reading everything with everything.

What observability do we get?

Every model call instrumented with OpenTelemetry GenAI semantic conventions — model, tokens in and out, latency, cost, parent span — exported into whatever your on-call team already runs, so your SRE sees a model span next to a service span in their own tool. Prompt-level tracing hosted or self-hosted depending on your residency rules. Per-tenant spend metering. PII redaction before anything is logged. And trace replay, so any production trace can be re-run against a new prompt or model to see the diff before the change ships rather than after.

Bring an architecture question

The first call is usually an engineer asking why we would choose one thing over another for their case. Bring the constraint that makes your problem awkward — the residency rule, the legacy surface, the latency budget — and we will tell you what we would build and what we would refuse to promise.

Talk to an Engineer Contact Us

Our second practice

This is our AI engineering practice

It is real work and it is where our four products came from. But what Cognilium leads with is narrower: optimization apps that run in tandem with Microsoft Dynamics 365, computing the decisions the ERP records but does not derive — the optimal price, the optimal pick path, the optimal stock level. See the optimization apps · How we build inside the ERP.