Why do recurring integrations create duplicate documents?

A duplicated document in Dynamics 365 after an inbound feed is rarely a lost message and rarely a defect in the ERP. It is the queue doing what Microsoft documents it to do: recurring integrations re-serve any message you never acknowledged, on a fixed interval, until you acknowledge it.

The consumer worked. It read the document, wrote it, and failed on the one call that has nothing to do with the document — and the queue, correctly, offered the same message again. Here is the contract in Microsoft's words, the three ways teams break it, one more path that re-serves the same message, one upgrade that breaks your defence against it, and what will not fix any of them.

One scope line first. Recurring integrations [GA] is a cloud feature — under Authorization for the integration REST API, Microsoft's Recurring integrations page states: "This feature isn't supported with Dynamics 365 Finance + Operations (on-premises)." On-premises the bulk path is the package REST API [GA] instead.

The mechanism, in one sentence

From a Note under API for acknowledgment on the Recurring integrations page:

"Until a message is successfully acknowledged, the same message becomes available to dequeue every 30 minutes. In cases when a message is being dequeued more than one time, the dequeue response sends the last dequeued date time. This date is blank for the first dequeue of a message. To prevent repeated downloads of the same message, make sure you acknowledge the message successfully. If an acknowledgment fails, include retry logic to handle the failure."

Read the second sentence again. It is the one that hands you a defence. The dequeue response tells you whether you have seen this message before — blank on a first delivery, populated on a redelivery. A consumer that reads it can refuse to write the document twice.

Our reading: the interval is a property of the queue. Neither that page nor the Data import/export framework parameters page publishes a setting that changes it — those are the two pages we searched, and the parameters page's Recurring integrations compatibility group carries one parameter, which is not a timer.

The acknowledgement contract

Three endpoints, and the risk sits entirely in the third.

Import (enqueue) — Verb: POST · URL: https://<base URL>/api/connector/enqueue/<activity ID>?entity=<entity name>
Export (dequeue) — Verb: GET · URL: https://<base URL>/api/connector/dequeue/<activity ID>
Acknowledgment — Verb: POST · URL: https://<base URL>/api/connector/ack/<activity ID>

The activity ID is not invented either: "To get the activity ID, on the Manage scheduled data jobs page, in the ID field, copy the globally unique identifier (GUID)."

Then the sentence that decides whether your integration duplicates: "You must include the body of the response from /dequeue in the body of the /ack POST request." An acknowledgement is an echo, not a receipt.

We log the ack request and its response verbatim, because the page states the requirement and not what a wrong body returns.

Two operations let you ask the ERP what it thinks happened rather than infer it: GetMessageStatus, whose documented values include Enqueued, Dequeued and Acked, and GetExecutionIdByMessageId, which takes the enqueued message ID and returns the execution ID.

Three ways the loop breaks

The ack is never sent — What the ERP sees: A message dequeued and not acknowledged · What you see: The same document again, on the queue's interval, indefinitely
The ack carries the wrong body — What the ERP sees: The same · What you see: The same — plus a middleware log full of successful HTTP calls
The loop was proven where one of its failure modes cannot occur — What the ERP sees: Nothing · What you see: Behaviour in production the sandbox never showed

The third row is Microsoft's, under HTTP vs HTTPS: "The dequeue API returns HTTP instead of HTTPS. You can see this behavior in application environments that use a load balancer, such as production environments. You can't see the behavior in a one box environment."

The download path your ack loop depends on behaves one way where it was built and another way where it matters, and Microsoft says so in a section heading.

One more redelivery path, and one broken defence

A stuck message is returned to the queue. On the framework parameters page, under Compatibility options > Recurring integrations compatibility, the parameter Skip reprocessing in process recurring integration messages reads: "Select No to automatically update the status of stuck messages from Processing to Queued during the next execution."

A message in flight when its batch restarted is processed again. Whether the first pass already wrote anything is the whole question, and the parameter does not answer it — which is why we treat the document's own natural key as the last line of defence rather than the queue's state machine.

An upgrade moves the batch job your custom code attached to. This one does not re-serve anything; it breaks whatever you built to stop the redelivery.

The recurring integrations page documents a change "available starting from PU64": "There's one regular batch job (Job1) that creates a new runtime child job (Job2). The regular batch task is added to Job2 instead of Job1."

The Note is explicit about who this hurts: "if you created your own custom batch task and add the task to Job1 as per the previous design, you're adding tasks to the wrong job."

Code that customizes SysIntegrationActivityBatch or SysIntegrationActivityBatchTask is a pre-upgrade review item — Microsoft states you "might encounter problems with the recurring integrations feature under the new design" — and the equivalent restructuring for import and export batch jobs is documented separately, "starting from version 10.0.42", against different classes.

What will not fix it

Ordering. Process messages in order is real, and it is not de-duplication: "You can enable this option to force sequential processing of incoming files in an import scenario. This option is only applicable to files and not data packages." Sequential delivery of the same message twice is still the same message twice.

Throttling. If your first hypothesis is that the ERP rejected the ack under load, check the scope. The Service protection API limits page lists services that are "currently exempt from them", and both Recurring integrations and Data Import/Export Framework (DIXF) are on it.

The same page adds that the exemptions are not permanent: "the documentation is updated when exemptions are removed for these services."

Where this bites hardest: inbound EDI

Look at what the three documented endpoints are keyed on. In the API for import (enqueue), API for export (dequeue) and API for acknowledgment sections, the identifiers are the activity ID and the message.

No identifier from inside the payload appears in any of those three sections — no envelope, no partner, no control number. The queue's identity is the message. The document's identity is inside it, and the protocol never looks.

For a customer-group load, fine. For an inbound purchase order from a trading partner it is the incident: a redelivered message becomes a second sales order carrying the same partner reference.

Which is why this article and What can't Dynamics 365 do natively for EDI? are one argument in two halves.

This half tells you why the same document arrives twice. That half takes up the envelope layer — partner identity, control numbers, acknowledgement of the exchange itself — which is exactly where the de-duplication key has to live, because the queue will not carry it for you.

Where we would draw the line

We would not write the acknowledgement loop in X++ inside Finance and Operations. Retry, dead-letter and replay are middleware concerns with an operational surface, and the ERP is the place an integration team can least observe them.

We would not treat de-duplication as a data-cleanup exercise. If the only defence is someone noticing, the defence is a person.

And we would not let an optimizer inherit this: Cognilium's apps surround Dynamics rather than touching its core, so our write path owns its own idempotency key and its own replay record. The queue's state machine is something we observe, never something we trust for identity.

What to do this week

Log the last dequeued date time from every dequeue response. Microsoft states it is blank on a first delivery, so you already have a redelivery flag you are probably discarding.
Log the /ack request and response bodies verbatim for one day, and confirm the request body is the dequeue response rather than a re-serialised version of it.
Record the value of Skip reprocessing in process recurring integration messages at Data Management > Framework parameters > Compatibility options > Recurring integrations compatibility. Record it; do not change it in production first.
Search your codebase for SysIntegrationActivityBatch and SysIntegrationActivityBatchTask. Any hit is a pre-upgrade review item against Microsoft's Job1 and Job2 note.

About Cognilium Cognilium builds AI optimization apps for Microsoft Dynamics 365 — companion apps that optimize the pricing, inventory, warehouse and planning decisions your ERP manages but can't optimize. Dynamics is your system of record. Cognilium is your system of intelligence. https://cognilium.ai · https://www.linkedin.com/company/37180269/

If duplicates are showing up downstream of a recurring data job, book a fifteen-minute call and we will walk your dequeue-and-ack path with you on your own estate — no deck. https://cognilium.ai

Why do recurring integrations create duplicate documents?