Skip to main content

IR 3.1.0 and v3 document trees

Status: Implemented in morphir-rust (finos/morphir-rust#256 and #257) and in the morphir CLI, and recorded in decision 0018. The normative text is now Document Tree File Formats (Version 3), the 3.1.0 section of What's New in Version 3, the v3 JSON Schema, and the format-version page. This draft stays as the design record. Where it and the implementation differ, the implementation and the normative pages win:

This draftImplementation
Does not say how a reader treats a Specs declared below 3.1.0Every reader (JSON, YAML, Ion) refuses it with specs_before_3_1
Does not say what a Specs body may holdIts own modules must be module specifications; a definition-shaped module is refused, and an Ion kind: specs datagram refuses a definition with definition_in_specs
Does not say what migration does with a SpecsA v3 Specs migrates to a v4 Specs with "formatVersion": 4, as a Library does
A v3 manifest has no entryPointsIt has no member beyond formatVersion, distribution, package, pathBudget and dependencies: entryPoints, version, created and layout are unknown_member, and a module manifest does not accept the v4 alias module
Does not say what a dependency may nameA dependency naming the distribution's own package is invalid_distribution_shape; a repeated one is duplicate_member
read_any_tree dispatches on the manifestAlso, the v4 tree reader refuses a manifest of major 3 with version_mismatch at manifest#/formatVersion
A tree of another 3.x release fails with unsupported_format_version_minorOnly a single-file document gets that diagnostic. The v3 tree reader accepts a manifest of exactly "3.1.0" and refuses any other 3.x release with version_mismatch at manifest#/formatVersion (finos/morphir-rust#262 tracks whether it should answer the minor diagnostic instead)
Any v3 document tree says "3.1.0"Only a JSON or YAML tree. The draft Ion tree follows the Ion draft, where a v3 library tree still says "3.0.0"
The TypeScript binding runs the kit cases as pendingIt skips them ("version 3 not in capabilities") until finos/morphir-typescript#36 lands
The Ion spelling writes own modules as module::specA repeated own module::spec is refused with duplicate_name; own modules do not merge
TreeParts<P> holds a tree's partsThe kit has a Payload trait and a TreeModel trait with associated file types; there is no TreeParts type
morphir ir migrate writes v3 JSON and YAML treesmorphir migrate (also morphir ir migrate) --target-version v3 --output-layout vfs writes v3 trees with --output-format json, yaml or ion; in every tree format the manifest decides the version on read

This draft adds a minor revision of the classic IR, 3.1.0. The revision adds two things. A v3 distribution may be a Specs distribution, and a v3 distribution may be stored as a JSON or YAML document tree. Before this revision, only the Ion tree held v3 (issue 970).

A 3.0.0 document stays valid and keeps its meaning. A writer emits the lowest version that expresses its content. A reader that implements this draft accepts [3.0.0,3.2.0).

Version rules​

ContentformatVersion a writer emits
Single-file v3 Library3 (the 3.0.0 release), unchanged
Single-file v3 Specs"3.1.0"
Any v3 JSON or YAML document tree"3.1.0" in every tree file

In a single-file document, a reader accepts the integer 3 and the release strings "3.0.0" and "3.1.0". 3.2.0 and later fail with unsupported_format_version_minor, as format-version support describes. The reference support table becomes [3.0.0,3.2.0),[4.0.0,4.1.0).

A binding that stays on [3.0.0,3.1.0) still reads every 3.0.0 document. It refuses a single-file Specs distribution with the minor-version diagnostic, which is the behaviour decision 0016 requires. A v3 tree manifest of any 3.x release other than "3.1.0" is version_mismatch (see the status note).

The Specs distribution​

A v3 Specs distribution publishes a package's specification without its definitions. It mirrors Library:

{
  "formatVersion": "3.1.0",
  "distribution": [
    "Specs",
    [["my"], ["pkg"]],
    [ [ [["morphir"], ["s", "d", "k"]], { "modules": [ … ] } ] ],
    { "modules": [ [ [["basics"]], { "types": [ … ], "values": [ … ], "doc": null } ] ] }
  ]
}

The third element lists dependency specifications, as Library does. The fourth element is a package specification: its modules are module specifications, spelled as a dependency's modules are.

The Ion spelling uses kind: specs in the morphir:: header and writes the distribution's own modules as module::spec values, as the v4 Ion spelling does.

Document tree files​

A v3 tree uses the v4 tree's logical paths, profiles, escaping, truncation and fileNames rules without change (document-tree files). Only the file contents differ. Every file carries "formatVersion": "3.1.0". A file whose version differs from the manifest's is rejected at that file.

Envelope members hold canonical strings, the same as the v4 envelope: package, path, name, dependencies, and the keys of fileNames. A canonical string comes from the classic word arrays, so [["morphir"], ["s", "d", "k"]] is morphir/SDK. A def or spec payload is the classic v3 JSON for that entry, exactly as a single-file document writes it.

Distribution manifest​

{ "formatVersion": "3.1.0", "distribution": "Library", "package": "my/pkg", "pathBudget": 4000, "dependencies": ["morphir/SDK"] }

distribution is Library or Specs. A v3 tree has no entryPoints. dependencies lists the deps/ packages in order and is omitted when empty. pathBudget is required, with the v4 floor of 64.

Module manifest​

{ "formatVersion": "3.1.0", "path": "domain/user", "doc": "User records.", "types": ["user"], "values": ["find"] }

Under pkg/ in a Library, access is written only when it is Private. A Specs tree under pkg/, and every tree under deps/, holds module specifications, which have no access. types and values list names. A reader also accepts inline entries keyed by canonical name, whose values are node payloads, as the v4 reader does. fileNames records cut stems, as in v4.

Type and value files​

{ "formatVersion": "3.1.0", "name": "user", "def": { "access": "Public", "value": { "doc": "", "value": ["TypeAliasDefinition", [], <Type>] } } }
{ "formatVersion": "3.1.0", "name": "int", "spec": { "doc": "", "value": ["OpaqueTypeSpecification", []] } }
{ "formatVersion": "3.1.0", "name": "find", "def": { "access": "Public", "value": { "doc": "", "value": { "inputTypes": [ … ], "outputType": <Type>, "body": <Value> } } } }
{ "formatVersion": "3.1.0", "name": "add", "spec": { "doc": "", "value": { "inputs": [ … ], "output": <Type> } } }

A Library tree holds def files under pkg/ and spec files under deps/. A Specs tree holds spec files in both. A def payload keeps v3's attribute slots, including a value's inferred types.

Implementation in morphir-rust​

The kit's layout keeps one implementation of the rules the compatibility kit pins, over a generic payload:

  • TreeParts<P> holds the manifest envelope and each module's root, package, path, access, doc, and (name, payload) for its types and values. A Payload trait lets the layout read formatVersion, name and listings from a file of type P, and build one. serde_json::Value implements it for the JSON and YAML profiles.
  • A TreeModel trait turns TreeParts into a distribution and back. The v4 model keeps today's decoders. The v3 model decodes and encodes the classic types.
  • read_tree and write_tree stay v4. read_tree_v3, write_tree_v3 and the per-module streaming writers are new. read_any_tree dispatches on the manifest's formatVersion.
  • The document-tree transport accepts v3 for every profile, streams v3 modules through the kit's writers, and reads a tree at the version its manifest names. A manifest whose version differs from the selected one is version_mismatch.
  • morphir ir migrate writes v3 JSON and YAML trees, and version detection reads any manifest. ir_storage::read_value reads any v3 tree. Compile output stays v4.

The Ion tree keeps its own layout code for now. A follow-up moves it onto TreeParts<ion_rs::Element>, so that the path, stem, budget and stray-file rules exist once for all three profiles.

Compatibility kit​

New cases, each marked version=3:

  • document-tree: a v3 manifest; a Library module with def files; deps/ with spec files; a Specs tree; a cut stem recorded in fileNames; the YAML profile of one tree; a v4 file inside a v3 tree, rejected.
  • versions: 3.1.0 accepted, 3.2.0 rejected.
  • distributions: a single-file v3 Specs distribution.

The Rust adapter declares these cases. The TypeScript binding skips them ("version 3 not in capabilities") until finos/morphir-typescript#36 lands.

Delivery​

  1. The kit refactor lands alone and changes no behaviour. Every existing layout, kit, transport and Ion tree test passes unchanged.
  2. 3.1.0 in the classic model: Specs, the version strings, the semantic header, the JSON, YAML and Ion single-file codecs, and the support table with its conformance corpus.
  3. v3 JSON and YAML trees in the kit and the transport.
  4. The CLI, this draft's normative pages (docs/spec/ir/schemas/v3/document-tree-files.md and the 3.1.0 whats-new), the v3 schema's Specs, the compatibility-kit cases, a kb Decision Record, and the changelog.

Out of scope​

  • TypeScript v3 trees and 3.1.0, and the 3.1.0 support-table raises in morphir-elm, morphir-scala and morphir-python, are follow-up issues.
  • Compile output stays v4. write_v3 keeps writing single files.
  • An application distribution has no v3 form.