Skip to content

Commit 7134490

Browse files
candleindarkcandleindark
committed
docs: consolidate vendor-configurable metadata models design docs into one file
1 parent 7c0737f commit 7134490

File tree

3 files changed

+154
-253
lines changed

3 files changed

+154
-253
lines changed

‎doc/design/vendor-configurable-metadata-models-1.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Vendor-Configurable Metadata Models
2+
3+
This document illustrates the limitations originate from the metadata models in the DANDI ecosystem with
4+
[dandi-schema](https://github.com/dandi/dandi-schema) before version 0.12.0 and outlines the solution
5+
of vendor-configurable metadata models implemented across multiple components of the ecosystem,
6+
with the implementation starting and coordinated with dandi-schema at v0.12.0, to overcome these
7+
limitations.
8+
9+
## Limitations
10+
11+
The current metadata models and their uses in the DANDI ecosystem have the following limitations:
12+
13+
1. We do not actually support or use multiple versions of the metadata models in dandi-archive.
14+
2. We use two representations of metadata models – Pydantic models and their respective JSON Schema derivatives – and
15+
rely on an external process to generate the JSON Schema representation from the Pydantic models.
16+
3. We manually trigger updates of web frontend files according to a specific version of the JSON Schema representation of the models.
17+
4. We hardcode vendor-specific information inside the dandi-archive codebase (backend and frontend).
18+
5. Any vendor-specific configuration done at runtime in Pydantic models is not reflected in the JSON Schema representation used
19+
by the web frontend since the web frontend uses a static versioned JSON Schema representation stored at
20+
[dandi/schema](https://github.com/dandi/schema) that has been generated by the external process mentioned in point 2.
21+
22+
```mermaid
23+
flowchart TD
24+
%% repositories as grouped nodes
25+
subgraph dandi_schema_repo["<a href='https://github.com/dandi/dandi-schema/'>dandi/dandi-schema</a>"]
26+
Pydantic["Pydantic Models"]
27+
end
28+
29+
subgraph schema_repo["<a href='https://github.com/dandi/schema/'>dandi/schema</a>"]
30+
JSONSchema["JSON Schema<br>serializations"]
31+
32+
end
33+
34+
subgraph dandi_cli_repo["<a href='https://github.com/dandi/dandi-cli'>dandi-cli</a>"]
35+
CLI["CLI & Library<br>validation logic<br/>(Python)"]
36+
end
37+
38+
subgraph dandi_archive_repo["<a href='https://github.com/dandi/dandi-archive/'>dandi-archive</a>"]
39+
Meditor["Web UI<br/>Metadata Editor<br/>(meditor; Vue)"]
40+
API["Archive API<br/>(Python; DJANGO)"]
41+
Storage[("DB (Postgresql)")]
42+
end
43+
44+
%% main flow
45+
Pydantic -->|"serialize into<br/>(CI)"| JSONSchema
46+
Pydantic -->|used to validate| CLI
47+
Pydantic -->|used to validate| API
48+
49+
JSONSchema -->|used to produce| Meditor
50+
JSONSchema -->|used to validate| Meditor
51+
Meditor -->|submits metadata| API
52+
53+
CLI -->|used to upload & submit metadata| API
54+
55+
API <-->|metadata JSON| Storage
56+
57+
%% styling
58+
classDef repo fill:#f9f9f9,stroke:#333,stroke-width:1px;
59+
classDef code fill:#e1f5fe,stroke:#0277bd,stroke-width:1px;
60+
classDef ui fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
61+
classDef data fill:#fff3e0,stroke:#e65100,stroke-width:1px;
62+
JSONSchema@{ shape: docs }
63+
64+
class dandi_schema_repo,schema_repo,dandi_cli_repo,dandi_archive_repo repo;
65+
class Pydantic,CLI,API code;
66+
class JSONSchema,Storage data;
67+
class Meditor ui;
68+
```
69+
> The diagram above depicts how metadata models are defined, represented, and used in the DANDI ecosystem before version 0.12.0 of dandi-schema.
70+
71+
## Solution to Overcome Limitations
72+
73+
The solution of vendor-configurable metadata model addresses the limitations outlined above by implementing the following changes:
74+
75+
1. Make the metadata models vendor-configurable in dandi-schema.
76+
2. Create an API endpoint at `/api/schemas/` in dandi-archive to distribute dynamically generated JSON Schema representations of the metadata models
77+
from the Pydantic models that define the metadata models at runtime.
78+
3. Update the `/api/info/` endpoint at dandi-archive
79+
1. to point to the endpoint in point 2 for retrieving a JSON Schema representation of the metadata models
80+
instead of a static versioned JSON Schema representation stored at [dandi/schema](https://github.com/dandi/schema).
81+
2. to include vendor-specific information.
82+
4. Remove any hardcoded vendor-specific information in dandi-archive, and use the vendor-specific configuration
83+
provided by dandi-schema.
84+
5. Have clients, such as the web UI (meditor), use the endpoints in point 2 and 3 to retrieve vendor-specific configurations
85+
and JSON Schema representations of the metadata models that match the metadata models used by the particular instance of dandi-archive.
86+
87+
88+
```mermaid
89+
flowchart TD
90+
%% repositories as grouped nodes
91+
subgraph dandi_schema_repo["<a href='https://github.com/dandi/dandi-schema/'>dandi/dandi-schema</a>"]
92+
Pydantic["Pydantic Models"]
93+
end
94+
95+
subgraph schema_repo["<a href='https://github.com/dandi/schema/'>dandi/schema</a>"]
96+
JSONSchema["JSON Schema<br>serializations"]
97+
98+
end
99+
100+
subgraph dandi_archive_repo["<a href='https://github.com/dandi/dandi-archive/'>dandi-archive</a>"]
101+
Meditor["Web UI<br/>Metadata Editor<br/>(meditor; Vue)"]
102+
API["Archive API<br/>(Python; DJANGO)"]
103+
SchemaEndpoint["Schema API Endpoint<br/>(JSONSchema)"]
104+
Storage[("DB (Postgresql)")]
105+
end
106+
107+
subgraph dandi_cli_repo["<a href='https://github.com/dandi/dandi-cli'>dandi-cli</a>"]
108+
CLI["CLI & Library<br>validation logic<br/>(Python)"]
109+
end
110+
111+
%% main flow
112+
Pydantic -->|"serialize into<br/>(CI)"| JSONSchema
113+
Pydantic -->|used to validate| CLI
114+
Pydantic -->|used to validate| API
115+
116+
API -->|serialize at runtime| SchemaEndpoint
117+
SchemaEndpoint -->|used to produce| Meditor
118+
SchemaEndpoint -->|used to validate| Meditor
119+
120+
Meditor -->|submits metadata| API
121+
CLI -->|used to upload & submit metadata| API
122+
API <-->|metadata JSON| Storage
123+
124+
%% styling
125+
classDef repo fill:#f9f9f9,stroke:#333,stroke-width:1px;
126+
classDef code fill:#e1f5fe,stroke:#0277bd,stroke-width:1px;
127+
classDef ui fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
128+
classDef data fill:#fff3e0,stroke:#e65100,stroke-width:1px;
129+
JSONSchema@{ shape: docs }
130+
131+
class dandi_schema_repo,dandi_cli_repo,dandi_archive_repo repo;
132+
class Pydantic,CLI,API,SchemaEndpoint code;
133+
class JSONSchema,Storage data;
134+
class Meditor ui;
135+
```
136+
> The diagram above depicts how metadata models are defined, represented, and used in the DANDI ecosystem starting with version 0.12.0 of dandi-schema.
137+
138+
139+
### Benefits
140+
141+
This implementation provides several benefits:
142+
143+
1. **Runtime Consistency**: The schema used by the frontend will always match the one used by the backend, including any vendor-specific configurations.
144+
2. **Simplified Deployment**: No need to manually update JSON Schema files or manage the [dandi-schema](https://github.com/dandi/dandi-schema) repository for storing
145+
JSON Schema representations of the metadata models.
146+
3. **Future-Proofing**: The implementation allows for future support of multiple schema versions if needed.
147+
4. **Reduced Dependencies**: Removes the dependency on external GitHub URLs for schema definitions.
148+
149+
### Immediate Steps and Longer-Term Opportunities
150+
151+
The immediate implementation supports distributing, through an API endpoint, the JSON Schema representations of only the vendor-configurable metadata models currently being used
152+
by the particular instance of dandi-archive. However, the API endpoint has been structured to allow support for multiple versions of JSON Schema representations in the future if needed.
153+
154+
Additionally, the JSON-LD context.json could also be similarly generated and served by the backend if needed in the future.

‎doc/design/vendored-schema-1-implementation.md‎

Lines changed: 0 additions & 46 deletions
This file was deleted.

0 commit comments

Comments
 (0)