Back to Blog
Published:
Last Updated:
Fresh Content
ERP Write-Back & IntegrationChapter 8

How do you move a million rows into Dynamics 365 without breaking it?

9 min read
2,106 words
high priority
Muhammad Mudassir

Muhammad Mudassir

Founder & CEO, Cognilium AI

TL;DR

The answer is the Data management framework's package REST API, in batch — and the value is four operational facts the call itself never tells you: the caller cannot parallelise it, parallel package import is a precondition, the blob file lives seven days, and error retrieval is two-phase and polled with an ambiguous empty response. Each verified against Microsoft's page, operation names character-exact.

How do you move a million rows into Dynamics 365 without breaking it?

Through the Data management framework's package REST API [GA], in batch, as data packages — and by getting right four operational facts the API call itself never tells you. The call is easy. The four facts decide whether the cutover lands, or lands twice.

Microsoft's own guidance does not start at a million rows either. From Optimize data migration: "Begin the optimization phase by using a subset of the data. For example, if you must import one million records, consider starting with 1,000 records, then increase the number to 10,000 records, and then increase it to 100,000 records."

Why this API and not the request-shaped ones

Microsoft's integration overview gives the threshold and hedges it in the same breath: "It's difficult to define what exactly qualifies as a large volume. The answer depends on the entity… However, here's a rule of thumb: If the volume is more than a few hundred thousand records, use the batch data API for integrations."

Its worked scenarios put figures on that — an inbound sales-order feed at "200,000 records per hour", an outbound purchase-order feed at "300,000 records per hour" — both landing on batch data APIs, and both carrying the page's caveat: "Use these numbers only to gauge the pattern and don't consider them as hard system limits."

Quote that line to anyone asking you to commit a cutover window from a documentation table.

Microsoft puts two batch data APIs side by side on the package API page, under Choosing an integration API: recurring integrations schedules inside the ERP and takes files or packages; the package REST API schedules outside it and takes only data packages. On a cutover you own the schedule, so it is the package API.

What you actually send

Not rows. A package — "A single compressed file that contains a data project manifest and data files", per Data management overview, or on Process and consume data packages, "a package manifest, a package header, and any other files for the data entities that are included."

It carries its own load order, sequenced by execution unit, level and sequence.

The data import and export jobs page states the semantics plainly — "In each execution unit, entities are processed in parallel", "After one level is processed, the next level is processed" — and names where your rows land first: "By default, the data import and export process creates a staging table for each entity in the target database."

That is why there are two error surfaces below rather than one.

The operations, named exactly

Character for character from Data management package REST API. Each is an OData action on DataManagementDefinitionGroups.

  • GetAzureWriteUrl — What it does: Returns a writable blob URL with an embedded SAS token; upload the package there · The detail that bites: "An SAS is valid only during an expiry time window."
  • ImportFromPackage — What it does: Starts the import; returns the execution ID, which "appears as the Job ID in the UI". ImportFromPackageAsync has "the same specifications" · The detail that bites: overwrite: "Always set this parameter to False when a composite entity is used in a package."
  • GetExecutionSummaryStatus — What it does: Status of a data project execution job · The detail that bites: Values: "Unknown, NotRun, Executing, Succeeded, PartiallySucceeded, Failed, Canceled"
  • GetExecutionErrors — What it does: "returns a set of error messages in a JSON list" · The detail that bites: Execution-level, not row-level
  • GetImportStagingErrorFileUrl — What it does: The "input records that failed at the source to the staging step of import for a single entity" · The detail that bites: "An empty string is returned if no error file is generated"
  • GenerateImportTargetErrorKeysFile — What it does: Creates the keys file for records that failed "during the staging to target step" · The detail that bites: Returns a Boolean, not a URL
  • GetImportTargetErrorKeysFileUrl — What it does: The URL of that file · The detail that bites: An empty string means two different things

One composite limit, in Microsoft's own grammar: "the limitation is that package that consists of a composite data entity must only have that one composite data entity."

Execute, then poll — and mind the empty string

ImportFromPackage returns fast, and that speed is not success: "ImportFromPackage() uses a batch to perform the import." The consequence, from the integration overview: "the status of the data import or export isn't pushed to the caller. The caller must poll for the result…"

Row-level failures are a second two-phase dance. Microsoft's Important box:

"Before you call this API, call the GenerateImportTargetErrorKeysFile API to generate the error file. If the GenerateImportTargetErrorKeysFile API returns true, call this API in a loop until it returns a non-empty string. If the GenerateImportTargetErrorKeysFile API returns false, this API always returns an empty string, because there are no errors."

The trap is one paragraph above it: "If the error file is still being generated, or if there isn't an error file, the API returns an empty string." An empty string means "no errors" or "not ready yet", and nothing in the response distinguishes them. A client that calls once and moves on drops every staging-to-target failure in the run, behind a green log line.

The two surfaces are different objects too: GetImportStagingErrorFileUrl points at the offending input records, GenerateImportTargetErrorKeysFile the keys of rows that failed staging to target. A row can parse perfectly and still be rejected by the target — which is why GetExecutionSummaryStatus has a value called PartiallySucceeded.

The parallelism trap

The instinct is to shard the load and fire the imports concurrently. Microsoft closes that door and names where the door is:

"ImportFromPackage() uses a batch to perform the import. Therefore, parallel processing rules must be used in Data management to perform parallel imports. Don't call ImportFromPackage() in parallel threads. Otherwise, it fails."

The parallelism is a setting inside Finance and Operations, not a property of your orchestrator. The path: System administration > Workspaces > Data management, the Framework parameters tile in the Import / Export section, the Entity settings tab, Configure entity execution parameters.

  • Import threshold record count — What it controls: Records processed per thread · Microsoft's own words: "If a file has 10,000 records, a record count of 2,500 with a task count of four means each thread processes 2,500 records."
  • Import task count — What it controls: Threads used for that entity · Microsoft's own words: The count "can't exceed the max batch threads allocated for batch processing in System administration >Server configuration"
  • Maximum batch threads — What it controls: Threads per Application Object Server instance · Microsoft's own words: "Currently, the default value is 8… don't set the field to a value that's more than 16 unless you do significant performance testing."

Those multiply: eight threads across four servers in the data migration batch group gives "the total maximum value of the Import task count field is 32 (= 8 × 4)".

Not every entity plays, and Microsoft prints the refusal verbatim: "Custom sequence is defined for the entity 'Customers V3', more than one task is not supported." Batch mode is the other precondition — "Run an import job in batch mode. Otherwise, a single thread runs the job."

Microsoft attaches a counterweight to all of this, and it is the half most summaries drop. The same Note that publishes the arithmetic warns that adding too many parallel tasks pins the infrastructure and degrades the environment.

It instructs: "Understand the resource capacity of the environment and consumption based on the parallel import tasks configured and limit the number of tasks." 32 is a ceiling, not a target.

So is a compatibility option. From Microsoft's recurring integrations page: "Parallel package imports by using recurring integration or package REST APIs are only supported if the Enhanced parallel package import option is enabled."

What none of the pages states is its default — we looked on the framework parameters page, the recurring integrations page, the package REST API page and the data import and export jobs page, and none of the four publishes one. Check the switch in your own environment.

The two clocks, and the on-premises sentence

Seven days on the blob. "The file in Blob storage stays for seven days. After that period, it's automatically deleted." The recurring integrations page states the same fact in different words — its own duplicate-document trap is a separate article. That is your replay window: the window in which the exact bytes you sent can still be sent again.

Ninety days on the job history. "By default, the system automatically deletes job history entries and related staging table data that are older than 90 days" — and the cleanup job archives what it purges, a feature Microsoft says has been "always on" since version 10.0.20.

Changing how long that archive survives, and downloading it, are the parts the same page calls "currently in preview" [PP].

Both clocks argue one thing: what you sent belongs in your storage, not the ERP's.

And if your deployment is Finance + Operations (on-premises), this API is not one option among several. Microsoft's integration overview says so in a Note directly beneath its own table of six integration patterns: "For on-premises deployments, the only supported API is the Data management package REST API. This is currently available on 7.2, platform update 12 build 7.0.4709.41184."

That sentence carries "currently" — Microsoft's word, a statement of current scope — and its "only" is bounded by where it sits: a Note under a six-row pattern table on one page, dated 2026-03-09. We are quoting Microsoft's scope claim, not making our own.

The package API page adds the specifics: append "/namespaces/AXSF" to the base URL, authorize through "Active Directory Federation Services (AD FS)", and expect GetAzureWriteUrl to return "the URL that is abstracted to local storage".

One line matters for capacity planning on cloud: the service protection page lists Data Import/Export Framework (DIXF) among the services "currently exempt" from its limits. Your constraint on a bulk load is batch capacity, not a request budget — which is why every lever above is a batch lever.

Where we would draw the line, and what to do this week

We would not shard a package import across parallel callers; the fix is a setting, not a client design. We would not treat Succeeded as done, because the status that costs money is PartiallySucceeded — and which rows it hides is only legible once both error files have been retrieved.

And we would not build the replay store inside the ERP — seven days is Microsoft's blob window, and anything longer is ours to hold.

Which is the discipline we hold our own write path to. Cognilium's apps surround Dynamics rather than touching its core: when Demand & Inventory Optimizer writes a buffer back, it writes a package it still holds, against a recorded execution ID, with both error files pulled before the run is called complete.

Three checks, each producing a fact:

  1. Call GetExecutionSummaryStatus on your last three production imports and record whether any returned PartiallySucceeded.
  2. Read your client code for the GetImportTargetErrorKeysFileUrl call. No loop means the empty-string defect.
  3. Compare Import task count for your heaviest entities against Maximum batch threads times the servers in your batch group, and record Enhanced parallel package import. A task count of one means single-threaded.

About Cognilium Cognilium builds AI optimization apps for Microsoft Dynamics 365 — companion apps that optimize the pricing, inventory, warehouse and planning decisions your ERP manages but can't optimize. Dynamics is your system of record. Cognilium is your system of intelligence. https://cognilium.ai · https://www.linkedin.com/company/37180269/

If you are designing the write path for a bulk load into Dynamics, book a fifteen-minute call and we will walk the package, the polling loop and the replay window with you on your own estate — no deck. https://cognilium.ai

Sources

Sources

Share this article

The work behind this series
AI optimization apps for Dynamics 365

The write-back mechanism every optimizer shares — read the system of record, compute the decision, return it behind an approval step.

Muhammad Mudassir

Muhammad Mudassir

Founder & CEO, Cognilium AI

Mudassir Marwat's argument is that ERP systems record decisions they never optimise.

Founder & CEO of Cognilium AI; 37 AI agents in production across four products; 4 production AI products built and operated; three clouds in production (AWSGCPAzure)
Agentic AIRAG → GraphRAG retrievalVoice AIMulti-Agent Orchestration
ContactView Profile
Next in this series
Is BYOD retired?
Chapter 9 · 8 min
In short

Key takeaways

  • The package REST API is asynchronous by consequence, not by decoration: the import call schedules a batch job and returns an execution ID, so the status of the import is never pushed back and the caller must poll for the result.
  • A caller cannot parallelise the import. Microsoft states that calling the import operation in parallel threads fails, and that parallelism is configured through parallel processing rules inside Data management instead.
  • Parallel package import is a precondition, not a default you can assume. Microsoft states that parallel package imports are supported only when the Enhanced parallel package import option is enabled, and its default position is not published on any of the four Microsoft pages this article searched.
  • Error retrieval is two-phase and polled, and an empty response is ambiguous by design — it means either that no errors exist or that the error file is still being generated. A client reading the first empty response as success drops the failures.
  • The uploaded package is deleted from blob storage after a fixed window, which bounds how long a run can be replayed from the ERP's copy rather than your own.
What goes wrong

Common mistakes to avoid

  • Sizing a cutover window from the volume figures in Microsoft's integration guidance. The page states those numbers are only for gauging the pattern and are not hard system limits.
  • Treating a completed status as a completed load. The summary status has a partial-success value, and the row-level failures live in two separate error files with two different shapes.
  • Checking only the source-to-staging errors. A row can parse cleanly into the staging table and still be rejected on its way to the target entity, and that is a different file retrieved by a different operation.
  • Assuming your orchestrator's thread count is the throughput lever. On this path the levers are the entity execution parameters, the batch threads per server and the servers in the batch group.