How to evaluate legal AI for an in-house team: seven questions that separate the real ones

Every legal-AI product demos well. That is the problem. A polished demo on a clean contract tells you almost nothing about whether the thing will survive contact with your actual work, your actual systems, and your actual team.

Here are the seven questions that do. They are neutral — they apply to any vendor, us included — and they are ordered by how often the answer quietly kills a project after purchase.

1. Does it review against our playbook, or against general legal knowledge?

A general model tells you what is standard. Your team does not care what is standard; it cares what is standard for you — your positions, your red lines, the playbook that took years to write. Ask to load your own playbook and see the review run against it. If the tool only knows "the market," it will flag the wrong things and miss yours.

2. Can it show its work?

For every flag, can the tool point to the exact clause and the exact rule that raised it? An answer with no source is a guess wearing a confident tone, and no lawyer will act on it without re-checking — which defeats the point. "Evidence over assertion" is not a nice-to-have in legal work; it is the whole job. Demand the citation, every time.

3. Does a person stay in control?

The right shape is: the system does the reading, extraction and first-pass flagging; the lawyer makes the call. Be suspicious of anything that positions itself as replacing judgement rather than removing the work that never needed judgement. Ask exactly where the human decides — and make sure the answer is "on everything that matters."

4. Where does it work — and will your team actually open it?

The most expensive legal AI is the one nobody opens. A tool that lives outside the systems your team already uses gets tried for a week and abandoned, because it asks everyone to switch context all day. Ask: does the review happen where the lawyer already works — in Word, in the contract system, in the ERP the obligations are tied to? Adoption is where most of these projects really die.

5. Where does our data live?

For regulated, privileged or sensitive agreements, this is not a preference — it is a requirement. Ask whether the agreements have to leave your environment, and if so, where they go and who can see them. The strongest answer is that the capability runs inside your own tenancy and the paper never leaves it.

6. Does it connect to the systems the obligations belong in?

A contract is often not the end of the job — it is the trigger for one. A renewal date needs to become a field on the record; an obligation needs to sit against the vendor it binds. Ask whether the tool can push what it extracts into your ERP or CRM, or whether it just produces another document you then re-key by hand. This is the line between a document tool and a decision tool. (It is also the specific gap we build for — legal AI that lives in the ERP.)

7. Buy or build — which does your situation actually call for?

Not every team should build. If your work is document-first — drafting, research, review, and the output is a document — a horizontal platform is likely the faster, cheaper, right call, and a good vendor will tell you so. If the contract has to update your operational systems, and your team won't adopt a second place to work, building the capability into the ERP tends to win. The honest test is one question: is the contract the deliverable, or the trigger for work in another system? (We wrote the long version of this test separately — build or buy legal AI — and we will tell you to buy when buying is right.)

Use these on us too

Point all seven at any vendor, including Cognilium. Our own product, Paralegent AI, reviews against your playbook, cites the clause behind every flag, keeps a person in control, and runs inside your environment — because those are the questions that decide whether legal AI gets used or shelved. If you are evaluating and want a straight second opinion, tell us what you run and what you are weighing.