Mega-issue: Vendorization of the dandi-schema and other components

[yarikoptic]

This issue provides an upper-level view of the plan and progress to make metadata models in the DANDI ecosystem vendor-configurable organized by release stages.

Release Stage 1:

Changes in this stage even if merged & released do not effect existing setup/functionality.

PRs could be merged independently.

• dandi-archive
  • Local runtime schema serialization endpoints dandi-archive#2386 (status: to be merged)
• dandi-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance dandi-infrastructure#224 (status: to be merged)
• linc-archive-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance lincbrain/linc-archive-infrastructure#24 (status: to be merged)
• ember-dandi-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance aplbrain/dandi-infrastructure#47 (status: to be merged)

Release Stage 2:

Changes, if merged and released, would affect functionality and provide vendorization for the archive instances.

Components must have coordinated release(s).

• dandi-schema: release first
  • Vendor-Configurable Metadata Models #294 (status: to be merged)
• dandi-archive:
  • Have the api/info/ endpoint provide instance config info dandi-archive#2430 (status: to be merged)
  • De/re-vendorize dandi-archive dandi-archive#2382 (status: to be addressed)
    • Utilize vendor specific info from schema instance config dandi-archive#2584
  • upgrade to the released dandi-schema

Stage X: good to have etc

Could be done now or later conditional on available dandi-schema release

• dandi-cli
  • Add vendor-specific behavior to dandi validate and `dandi upload, validation - add support for vendorization record from the instance server dandi-cli#1625
  • Make the dandi service-scripts publish-dandiset-version-doi command DANDI instance specific, incorporating vendor information of a particular DANDI instance dandi-cli#1704 (status: WiP)
    • Vendorize dandi-cli per config in server info dandi-cli#1661 (status: to be revived)
• dandi-archive
  • Along with manifests, dump (vendorized) schema + context to S3 dandi-archive#2577

[candleindark]

@yarikoptic Two issues we need to attend to:

• 1. Is there any plan for continue use of https://github.com/dandi/schema after the vendor-configuration effort? In Local runtime schema serialization endpoints dandi-archive#2386, we have replaced the need of https://github.com/dandi/schema for the distributing dandiset.json. However, dandi-archive still needs the context.json to be hosted at https://github.com/dandi/schema unless additional changes are provided.

  https://github.com/dandi/dandi-archive/blob/31fac2f78c75b9ee5708fabaaee02d417300bd4a/dandiapi/api/models/asset.py#L262-L265

  https://github.com/dandi/dandi-archive/blob/31fac2f78c75b9ee5708fabaaee02d417300bd4a/dandiapi/api/models/version.py#L234-L237

• 2. https://github.com/dandi/dandi-archive/blob/master/web/src/types/schema.ts is generated by running https://github.com/dandi/dandi-archive/blob/master/web/src/migrate.ts on a dandiset.json. We need to check if the generated schema.tss for different vendor-specific dandiset.jsons are the same. If they are different we may have a problem.

  • A possible way to do this to run yarn migrate on different vendor-specific dandiset.json

[yarikoptic]

1. I think solution would be at the level of dandi-archive dumps Along with manifests, dump (vendorized) schema + context to S3 dandi-archive#2577
2. If that schema.ts is used only for development, since it already includes instance-configuration specific metadata (such as licenses https://github.com/dandi/dandi-archive/blob/master/web/src/types/schema.ts#L205C1-L205C61 ) - such file then should be specific to the instance if someone would be developing using it. @mvandenburgh @jjnesbitt -- is that file used only for development or somehow also instructs front end at run time? (how is it used BTW -- can't see from quick search)