Import package format (kandra-import/1)

One JSON format carries both a configuration's demo data and real data imports. The import engine (IDataImportService, in Kandra.DataImport) applies a package step by step through the real CRUD services, and stamps every row it writes so that applying the package again is safe. For how to use it (entry points, a data processor that imports files, demo seeding) see the practical guide Data import and demo data.

A complete example

{
  "$schema": "./acme.import.schema.json",
  "format": "kandra-import/1",
  "package": "acme-demo",
  "version": 3,
  "origin": "demoSeed",
  "configuration": "Acme",
  "title": "Acme demo company",
  "steps": [
    { "dictionary": "Warehouses", "match": ["code"], "items": [
      { "$id": "wh-main", "code": "WH-MAIN", "name": "Main", "branchId": { "$code": "BR-1" } } ] },
    { "dictionary": "Counterparties", "items": [
      { "$id": "sup", "isFolder": true, "code": "SUPPLIERS", "name": "Suppliers" },
      { "$id": "sup-1", "code": "SUP-1", "name": "Roshen", "parentId": { "$ref": "sup" } } ] },
    { "constant": "OrganizationName", "$id": "org-name", "value": "Demo LLC" },
    { "document": "GoodsReceipts", "submit": true, "items": [
      { "$id": "gr1", "date": "$today-23d", "counterpartyId": { "$ref": "sup-1" },
        "toWarehouseId": { "$ref": "wh-main" },
        "lines": [ { "itemId": { "$code": "ITEM-1" }, "quantity": 200, "price": 12.5 } ] } ] },
    { "createFrom": { "source": "gr1", "target": "Waybills" }, "$id": "wb1", "submit": true, "set": { "code": null } },
    { "dataProcessor": "ImportNbuRates", "$id": "nbu", "run": "once", "optional": true, "input": { "date": "$today-30d" } }
  ]
}

Header

Field	Required	Meaning
$schema	no	Path or URL of the JSON Schema, for editor completion. Ignored by the engine.
format	yes	Always "kandra-import/1".
package	yes	The package id, matching ^[a-z0-9][a-z0-9._-]{0,199}$. It becomes the Source of every row's import stamp, so it identifies the package's rows for good.
version	yes	Non-negative integer. Bump it when the content changes. A demo installation re-applies its demo package when the embedded version is newer than the applied one.
origin	yes	"demoSeed" or "import". Stamped on every row and on every change event the run produces.
configuration	yes	Must equal the host's configuration name (the one passed to AddKandraDataImport("Acme")).
title	no	Shown on the admin page.
steps	yes	Array of steps, applied strictly in order.

Steps

A step's kind is the key it carries. Exactly one of these keys per step:

Step	Key	Other fields	What it does
Dictionary	"dictionary": "<catalog name>"	items, match, policies	Inserts or updates each object in items.
Document	"document": "<catalog name>"	items, submit, match, policies	Same, for documents. "submit": true inserts them submitted (isActive: true), so they post.
Constant	"constant": "<constant name>"	$id, value, policies	Sets one constant to value.
Create from	"createFrom": { "source": "<$id>", "target": "<document name>" }	$id, set, submit, onDeleted	Opens a new target document from an earlier document of this package (the same prefill as the UI's "Create …" button), applies set, inserts it.
Data processor	"dataProcessor": "<processor name>"	$id, input, run, optional	Executes the processor with input.

Catalog names are the Name of each Dto's [KandraDictionaryForm] / [KandraDocumentForm] / [KandraDataProcessorForm] attribute (for example Warehouses, GoodsReceipts). Constants use the Name of their [KandraConstant].

Items

• An item is the Dto's JSON wire shape (camelCase property names, as in the API and over MCP) plus a required $id. Unknown properties are errors.
• A [Required] property must be present, even if the Dto has a default. Exceptions: id; on documents, code (numbering supplies it when omitted) and isActive (set by submit).
• Documents: omit code to get the next number. Tabular lines (lines) are arrays of line objects; a line's id may be omitted. On update, a line keeps the id of the existing line at the same position.
• Hierarchical dictionaries: write "isFolder": true on folders (the engine converts it to the wire discriminator). Folder and leaf properties are both allowed and checked against the kind the item is. Put folders before the leaves that reference them.

createFrom

The source is the $id of a document earlier in the package. The engine calls the target's New with that source document (Document Links, [SourceFor]), lays set over the result and inserts it. In set, "code": null keeps the number New produced. A document created this way is never updated by a re-run: changing the step reports SkippedDrift (or SkippedUserModified), because updating would discard what New copied across.

dataProcessor

• run: "once" (default) executes the processor the first time the package is applied and records that in the package state; later runs report AlreadyRun. "always" executes it on every run.
• optional: true turns a failure into a Warning outcome instead of Failed. Use it for a step that needs something outside your control, such as a public web service.
• Rows the processor creates are not stamped: they belong to the processor's own logic, not to the package.

Identity: $id

$id is required on every item and on every constant, createFrom and dataProcessor step. It must be unique within the package and becomes the Key of the row's import stamp. Together with package it is how a later run finds the row again, so never rename a $id once a package has been applied anywhere: the engine would treat it as a new object. Derive it from a stable business key ("item:ITM-TEA"), not from a row number.

References

A reference field (a Guid that points at another entity) takes a plain id, or one of two reference objects:

Form	Resolves to
{ "$ref": "<$id>" }	An object defined earlier in this package. Forward references are errors.
{ "$code": "<Code>" }	An existing row of the referenced entity, found by Code. The entity is known from the field's metadata (its [Search] / [Dropdown] / [Dialog] source, or for parentId the dictionary itself). An object inserted earlier in the same run is found too.

A reference object must contain exactly one of the two keys. $code on a field that doesn't reference an entity is an error. A $ref to an object that failed or was skipped earlier in the run makes the referencing object fail.

Date tokens

A string value starting with $ is a token. Valid tokens:

Token	Meaning
$now	The current instant (UTC).
$today	The caller's local date.
$today-23d, $today+14d	That many days before or after it (up to 5 digits).

"Caller" is the user running the import: the administrator on the admin page, the Run As user and time zone on a scheduled job. A DateTime field gets local midnight of that date converted to UTC; a DateOnly field gets the local date. Tokens resolve when the package is applied but are hashed unresolved, so a package with tokens isn't seen as changed from one day to the next. Any other string starting with $ is rejected: a literal value like "$100" can't be written.

match: adopting existing rows

"match": ["code"] (any Dto properties) on a dictionary or document step tells the engine what to do when it has no stamped row for an object: look for an existing row without an import stamp whose matched properties equal the object's values. If exactly one qualifies, the engine adopts it: stamps it and brings it in line with the package according to onChange (outcome Adopted). If several qualify, the object fails as ambiguous. Without match, an object with no stamped row is always inserted.

Policies

Three policies decide what happens to an object whose row already exists. Each can be set per step:

Policy	Applies when	Values	Default
onChange	The object changed in the package since the last apply, and no user touched the row	update, skip, error	update for dictionaries and constants; skip for documents
onUserModified	The object changed in the package, and a user modified the row after it was imported	skip, overwrite, error	skip
onDeleted	A user soft-deleted the row	skip, restore, error	skip

• Updating a document (onChange: update) saves it with isActive set from submit: a submitted document is re-posted with the new content in the same save, a draft stays unposted.
• restore is not offered for constants; a deleted constant row is set again by the next import that changes it.
• error fails the object with a message naming the policy.

"Modified by a user" means the row's Modified time is later than its stamp's ImportedAt. The engine's own writes don't count.

Canonical hash

SourceHash on the stamp is the SHA-256 (lowercase hex) of the object's canonical JSON as written in the package: properties sorted by ordinal name at every level, no insignificant whitespace, numbers exactly as written, strings re-encoded uniformly, $ref / $code objects and date tokens unresolved. Items are hashed individually; constant, createFrom and dataProcessor steps are hashed as a whole. Consequences:

• reformatting or reordering properties doesn't change the hash;
• 12.5 and 12.50 are different numbers as written, so changing one into the other is a change;
• a referenced row changing doesn't change the referencing object's hash.

Validation and diagnostics

Before anything is written, the package is read and validated against your configuration's entity catalog. Problems are reported as diagnostics { path, severity, message }, where path locates the value (steps[2].items[0].lines[1].itemId). Errors include an unknown entity, constant or processor name, unknown properties, a missing required property, a duplicate $id, a forward or unknown $ref, $code on a non-reference field, a bad token, a match property that doesn't exist, and a configuration mismatch. Any error stops the run before the first write.

Then the engine plans the run (the whole of a dry run) and runs the licence pre-check: the documents the package would create, all counted in the current month, plus documents already created this month, against the monthly limit; and the longest line collection against the per-document line limit. A failure is an error at path $.

Report outcomes

Each object gets one outcome in the report (ImportReport.Items, with counts in ImportReport.Counts):

Outcome	Meaning
Inserted	No row existed; one was created.
Adopted	An unstamped row matched by match was taken over and brought in line with the package.
Updated	The object changed in the package and the row was updated (onChange: update, or onUserModified: overwrite).
Unchanged	The row's hash equals the object's. Nothing was written.
Restored	The row had been soft-deleted and was restored (onDeleted: restore).
Executed	A dataProcessor step ran.
AlreadyRun	A run: once dataProcessor step had run before.
SkippedUserModified	A user changed the row after import, and the policy kept their version.
SkippedDrift	The object changed in the package and onChange: skip left the row as it is (the default for documents).
SkippedDeleted	A user deleted the row, and onDeleted: skip left it deleted.
Warning	An optional dataProcessor step failed.
Failed	The object could not be applied; the message says why (a validator error, an authorization error, a missing $code, …).

ImportReport.HasErrors is true when there is an error diagnostic or a Failed object. With ContinueOnError: false (the default) the run stops at the first Failed object, and running again continues from there.

Package state and history

The engine keeps each package's applied version, the dataProcessor steps it has run and its last 20 real runs (time, user, counts and a link to the full report) in a reserved row of the constants table. The Constants admin page doesn't list that row. The admin page and GET api/v1/Import/runs read it. Dry runs aren't recorded there. The runs started through the admin page or the endpoints are also kept in memory for a while, dry runs included.

JSON Schema

The engine builds a JSON Schema (draft 2020-12) of your configuration's packages at runtime: every dictionary, document, constant and processor with its properties, the reference objects, tokens and policies. Get it from GET api/v1/Import/schema (administrators only), or use the snapshot the scaffolded configuration keeps next to its packages, Acme.Application/ImportPackages/acme.import.schema.json.

To get completion and validation in VS Code (or any editor using the JSON language server), point the package at the snapshot with a relative $schema:

{
  "$schema": "./acme.import.schema.json",
  "format": "kandra-import/1",
  ...
}

The snapshot is checked by a test in the scaffolded configuration (Tests/ImportPackages/DemoPackageTests), which fails when the snapshot no longer matches your entities. After adding an entity or a field, rewrite it:

UPDATE_IMPORT_SCHEMA=1 dotnet test

The schema checks shape (names, types, the $ref / $code objects, policy values). Reference targets, $id uniqueness and required fields per folder or leaf are checked by the engine's validation, so a dry run is still the final word.

Not in the format

• Users and passwords. Demo users come from the configuration's IDemoUserProvider, see Demo data.
• Register or ledger rows. Registers and postings are written only by submitting documents. Import the documents instead.
• Other file formats (CSV, bank statements, spreadsheets). Convert them into a package in a data processor; see Automated import is your own data processor.