Can you change an agent's prompt without a redeploy?

Can you change an agent's prompt without a redeploy?

Yes. And the interesting question is not whether you can — it is what you have given up in exchange.

A prompt is the behaviour of the system. Treat it as configuration and you gain the ability to fix things in seconds. You also lose the review, the diff and the rollback story that everything else governing behaviour gets for free.

For the engineering lead deciding whether prompts belong in the repository. 7 minute read.

Prompts are configuration, whether you planned it or not

Ship an agent system and this arrives within weeks. A category description is subtly wrong. A threshold is slightly too aggressive. An analyst's instructions produce the right judgement in the wrong tone.

None of those are code changes in any meaningful sense. The logic is correct. The values it operates on are not.

If prompts live in the repository, fixing any of them is a code change, a review, a build and a deploy — which is a long time to leave a customer with output that is subtly wrong. So most teams move them out, and having moved them, discover the second-order effects.

Three sources, one order

Our pipeline resolves configuration in a fixed priority order: the parameter store first, then a database table, then environment variables as a fallback.

That ordering is not arbitrary. The parameter store is the operational surface — change it and the change is intended. The database table carries structured per-environment configuration. Environment variables are what remains when neither is reachable, which matters most at startup and in local development.

What lives there is more than prompts. Agent definitions and prompt templates, category descriptions, scoring and routing thresholds, retry and backoff settings, quality filters, circuit-breaker parameters — all as typed models rather than loose strings, so a malformed value fails at load rather than at use.

That typing is doing real work. Configuration that can be edited by anyone in a console will eventually contain a mistake, and the only question is whether it surfaces immediately or three stages downstream in an agent's output.

Cached, refreshed, and deliberately stale

Reading configuration on every message would put the parameter store in the hot path of every model call. So it is cached with a five-minute lifetime and refreshed in the background.

And it is stale-while-revalidate: if the refresh fails, the cached values are served rather than raising an error.

That last choice is worth pausing on, because it is the opposite of what a strict system would do. A configuration service being unreachable does not stop the pipeline — workers carry on with the last good configuration they had. Given the alternative is halting a document mid-review because a parameter store had a bad minute, it is clearly correct.

It also means "the config is live" is not quite true. There is a window, bounded by the refresh interval, where different workers can hold different views of the world. For most settings that is harmless. For anything that must be applied consistently across a fleet, it is a real property to design around.

A long job must see one configuration

Here is the mechanism that makes hot-reload safe for work that takes minutes rather than milliseconds.

Configuration is snapshotted per job. A document entering the pipeline gets the configuration as it was at that moment, and that snapshot is used for the whole job.

Without it, hot-reload would be actively dangerous. Consider a contract in the middle of analysis when someone updates a threshold. The first forty chunks were scored under the old value and the remaining ten under the new one. The output is internally inconsistent in a way nothing in it records — a single report, two behaviours, no marker where the change happened.

That is a subtle and expensive class of bug, because the result looks complete. It is not an error, nothing failed, and the inconsistency is only discoverable by someone who knows a change was made and can correlate the timestamps.

Snapshotting turns configuration from a global variable into a job-scoped one, and it is the difference between hot-reload being a feature and being a hazard.

The governance problem

Now the part most teams meet later than they should.

A prompt change is a behaviour change. It alters what the system concludes about a customer's contract. Every comparably consequential change in your codebase goes through review, leaves a diff, lands in history and can be reverted by someone who was not involved.

A parameter-store edit has none of that by default. No pull request. No reviewer. No diff showing what the previous value was. No changelog entry. No test run against it. And frequently no answer to "who changed this, when, and what did it say before?"

The speed you gained is exactly the governance you lost, and they are the same property viewed from two sides. The reason a code change is slow is that several mechanisms are inspecting it.

This matters more here than in most systems, because the artefact being changed is closer to the product than a timeout value is. A prompt describing how to assess indemnity clauses is, functionally, part of the legal opinion the tool expresses — and it can be edited by anyone with console access, without anybody else seeing it happen.

Where it breaks

Environments drift, and drift is invisible. Once configuration lives outside the repository, staging and production diverge quietly. Somebody fixes production during an incident and does not backport it, so the next test run is against values that no longer describe reality.

A version hash helps and does not solve it. Our config carries one for change detection, which answers "has this changed?" — useful for cache invalidation and for correlating a behaviour shift with a moment in time. It does not answer "what changed, why, and who decided that." Those need a process, not a field.

And rollback is not automatic. Reverting a deploy restores code. It does not restore configuration, because configuration was never in the deploy. The rollback story for a bad prompt is somebody remembering what it used to say — which is why the previous value is the single most important thing to store and the thing most often lost.

What we would build, and the check for Monday

How we build it. Keep the hot path — it is right, and the alternative is leaving customers with broken output while a build runs. Then put the governance back deliberately: version every configuration change with its author, its previous value and a reason, and record the config version alongside every job result so any output can be traced to the exact behaviour that produced it.

The snapshot already exists per job. Recording which snapshot is a small addition and it is what makes an output explicable months later.

Status, stated plainly. This is a system we built. It runs, it is demonstrable on a call, and it has zero delivered engagements. We build these on request, against your systems. No number here describes anyone's business but our own.

The check worth running, and it needs nothing from us. Pick one prompt your system is using right now and answer three questions: what did it say last month, who changed it, and why? If you cannot answer all three, your most consequential behaviour is the one thing in your stack with no version control — and that is usually discovered while trying to explain an output to a customer.

About Cognilium Cognilium is an AI engineering company — agent systems, retrieval, knowledge graphs, voice AI and the production plumbing that makes them survive contact with real workloads. https://cognilium.ai · https://www.linkedin.com/company/37180269/

Want to know whether your prompts have a rollback story? Book a 15-minute call — we will walk your configuration path with you. No deck.

Sources

This article draws on our own build. The thresholds it describes are the subject of the sibling chapter on partial failure.