De/re-vendorize dandi-archive

[yarikoptic]

• NB Adapted from https://github.com/dandi/dandi-archive/blob/401108fb6f052ec9a71db39ed4bf415a67eb3366/doc/design/vendored-schema-1.md

Work is ongoing in dandi-schema:PR#294 to make vendorization of the schema configurable.
That would result in dandi/schema JSONSchema serializations becoming generally de-vendorized.
And it will be dandi-archive instance responsibility to re-vendorize, which would primarily consist in changing regular expressions more restrictive, via configuration/environment-variables.

ATM we still have some hardcoded vendorization in dandi-archive code:

Backend

Excluding some where we might want to vendorize too (e.g. email subjects etc):

❯ git grep DANDI: -- dandiapi | grep -v -e test_ -e 'subject=' -e 'verbose_name'
dandiapi/api/models/version.py:            'identifier': f'DANDI:{self.dandiset.identifier}',
dandiapi/api/models/version.py:            'id': f'DANDI:{self.dandiset.identifier}/{self.version}',
dandiapi/api/services/metadata/__init__.py:            f'DANDI:{publishable_version.dandiset.identifier}/{publishable_version.version}'
dandiapi/api/tests/fuzzy.py:DANDISET_SCHEMA_ID_RE = Re(r'DANDI:\d{6}')
dandiapi/api/views/dandiset.py:            if identifier.startswith('DANDI:'):

Web frontend

❯ git grep DANDI: -- web | grep -v -e test_ -e 'subject=' -e 'verbose_name'
web/src/components/DandisetList.vue:        DANDI:<b>{{ item.dandiset.identifier }}</b>
web/src/stores/dandiset.ts:      schema['properties']['identifier']['pattern'] = '^DANDI:\\d{6}$'
web/src/views/DandisetLandingView/DownloadDialog.vue:  // Use the special 'DANDI:' url prefix if appropriate.
web/src/views/DandisetLandingView/DownloadDialog.vue:  const dandiUrl = `DANDI:${identifier}`;

Should rely on information to come from API, hence:

• Have the api/info/ endpoint provide instance config info (replaced) #2396

It is worth going one by one and seeing where/how to adjust. E.g. this one would require input from @mvandenburgh since not sufficient information provided in change introduced in 433e351 (I think vue3 migration, could be wrong)

❯ git blame web/src/stores/dandiset.ts | grep -A2 TODO
433e351f4 (Mike VanDenburgh 2025-02-05 13:45:34 -0500 119)       // TODO: Fix this upstream in the schema
1f52c6445 (Mike VanDenburgh 2025-03-05 14:38:35 -0500 120)       // @ts-expect-error TS7053
433e351f4 (Mike VanDenburgh 2025-02-05 13:45:34 -0500 121)       schema['properties']['identifier']['pattern'] = '^DANDI:\\d{6}$'

Related

• Local deployment #2029
• Multi-instance DOI support complexity dandi-schema#307 -- we need to use proper config here if we don't yet

[candleindark]

To allow dandi-archive to revendorized based on environment variables, building on what you said above, I can think of changing all the currently hardcoded vendor-specified values to depend on the CONFIG we stablished in dandi/dandi-schema#294. Is that want you have in mind? That will be re-vendorizing only in the Python level and not the JSON-schema level in the backend. I presume the re-vendorization will be more limited in the frontend.

[yarikoptic]

Is that want you have in mind?

yes. But note that here we would need the value (e.g. DANDI) and not a regex/pattern (which could be DANDI but also could be .*). So may be we better define some explicit "instance_name" configuration item?

That will be re-vendorizing only in the Python level and not the JSON-schema level in the backend. I presume the re-vendorization will be more limited in the frontend.

That is what I have tried to figure out in the

• Design document to refactor access to JSONSchema serialization of dandi-schema's dandiset.json #2381

but I failed to find where/if we use jsonschema in frontend directly.

[candleindark]

Is that want you have in mind?

yes. But note that here we would need the value (e.g. DANDI) and not a regex/pattern (which could be DANDI but also could be .*). So may be we better define some explicit "instance_name" configuration item?

Good point. With that in mind, instead of grabbing both id_pattern, regex pattern for the prefix of identifiers and instance_name, I think we can just have the user input the instance_name for dandischema and share it with dandi-archive. instance_name will be used as the prefix of the identifier and as part of a regex pattern internally, but to the user there is no semantic of the input being a regex pattern. Doing so will make things simpler for the user, eliminate the logic requires to verify that the instance_name conforms to id_pattern, and consolidate information in one place. We can also apply the same principle to the datacite_doi_id_pattern config field.

In concrete terms, I want to replace the current definition of Config in dandischema.conf.py in dandi/dandi-schema#294.

current `Config`

from __future__ import annotations
from pydantic_settings import BaseSettings, SettingsConfigDict

class Config(BaseSettings):
    """
    Configuration for the DANDI schema

    Note
    ----
        Since this class is subclass of `pydantic.BaseSettings`, each field of an
        instance of this class can be populated from an environment variable of the
        same name prefixed with the prefix defined in `model_config` with the name
        of the environment variable interpreted **case-insensitively**.
        For details, see https://docs.pydantic.dev/latest/concepts/pydantic_settings/
    """

    model_config = SettingsConfigDict(env_prefix="dandi_schema_")

    id_pattern: str = r"[A-Z]+"
    """Regex pattern for the prefix of identifiers"""

    datacite_doi_id_pattern: str = r"\d{4,}"
    """
    The registrant code pattern of the DOI prefix at DataCite

    The number sequence that follows "10." within the DOI prefix as documented
    at https://support.datacite.org/docs/prefixes.
    """

with

proposed `Config`

from __future__ import annotations

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field


class Config(BaseSettings):
    """
    Configuration for the DANDI schema

    Note
    ----
        Since this class is subclass of `pydantic.BaseSettings`, each field of an
        instance of this class can be populated from an environment variable of the
        same name prefixed with the prefix defined in `model_config` with the name
        of the environment variable interpreted **case-insensitively**.
        For details, see https://docs.pydantic.dev/latest/concepts/pydantic_settings/
    """

    model_config = SettingsConfigDict(
        env_prefix="dandi_schema_", validate_default=False
    )

    instance_name: str = Field(default=r"[A-Z]+", pattern=r"^[A-Z]+$")
    """Name of the DANDI instance"""

    datacite_doi_id: str = Field(default=r"\d{4,}", pattern=r"^\d{4,}$")
    """
    The registrant code of the DOI prefix at DataCite
    """

Note that the pattern of each field ensures the input is of a particular pattern. When the input/environment variable is not provided, the default is a regex pattern. (The default values are not validated by the settings model.)

That will be re-vendorizing only in the Python level and not the JSON-schema level in the backend. I presume the re-vendorization will be more limited in the frontend.

That is what I have tried to figure out in the

• Design document with a diagram of metadata life cycle #2381

but I failed to find where/if we use jsonschema in frontend directly.

I am sure the frontend uses the JSON schema at dandi/schema directly. This line sets which JSON schema the frontend uses.

[yarikoptic]

Good point. With that in mind, instead of grabbing both id_pattern, regex pattern for the prefix of identifiers and instance_name, I think we can just have the user input the instance_name for dandischema and share it with dandi-archive. instance_name will be used as the prefix of the identifier and as part of a regex pattern internally, but to the user there is no semantic of the input being a regex pattern. Doing so will make things simpler for the user, eliminate the logic requires to verify that the instance_name conforms to id_pattern, and consolidate information in one place. We can also apply the same principle to the datacite_doi_id_pattern config field.

Overall I think it is sound, but I would prefer to make it so "by default" it would work for local installations. So instead of suggested

instance_name: str = Field(default=r"[A-Z]+", pattern=r"^[A-Z]+$")
"""Name of the DANDI instance"""

datacite_doi_id: str = Field(default=r"\d{4,}", pattern=r"^\d{4,}$")

what if we use

instance_name: str = Field(default=r"DANDI-ADHOC", pattern=r"^[A-Z][-A-Z]*$")
"""Name of the DANDI instance"""

datacite_doi_id: str = Field(default=r"", pattern=r"^(\d{4,}|)$")

thus making it we

• always have some instance name (DANDI-ADHOC) which we reserve for the default value if not vendorized
  • it would still match pattern/requirement to be non empty and capital
  • TODO: check if identifiers.org could have an identifier with - in the name. If "yes" - good", if "not" - let's change above to "DANDIADHOC" or alike and adjust pattern
• by default there would be no datacite_doi_id (empty). This should trigger total disablement of interactions with datacite (attn @asmacdo)

WDYT?

• Design document with a diagram of metadata life cycle #2381

but I failed to find where/if we use jsonschema in frontend directly.

I am sure the frontend uses the JSON schema at dandi/schema directly. This line sets which JSON schema the frontend uses.

That is where it is defined, but from what I gathered it is not used for anything besides reporting it in https://dandiarchive.org/server-info and https://api.dandiarchive.org/api/info/ . I would be happy to be corrected.

[candleindark]

• Design document with a diagram of metadata life cycle #2381

but I failed to find where/if we use jsonschema in frontend directly.

I am sure the frontend uses the JSON schema at dandi/schema directly. This line sets which JSON schema the frontend uses.

That is where it is defined, but from what I gathered it is not used for anything besides reporting it in https://dandiarchive.org/server-info and https://api.dandiarchive.org/api/info/ . I would be happy to be corrected.

I don't where exactly the location of the use of the JSON schema in the frontend is, but the frontend downloads the schema pointed by the backend's https://api.dandiarchive.org/api/info/.

dandi-archive/web/src/stores/dandiset.ts

Lines 109 to 132 in 430cc57

	async fetchSchema() {
	const { schema_url: schemaUrl } = await dandiRest.info();
	const res = await axios.get(schemaUrl);
	if (res.status !== 200) {
	throw new Error('Could not retrieve Dandiset Schema!');
	}
	const schema = await RefParser.dereference(res.data);
	// TODO: Fix this upstream in the schema
	// @ts-expect-error TS7053
	schema['properties']['identifier']['pattern'] = '^DANDI:\\d{6}$'
	// @ts-expect-error TS7053
	schema['$defs']['Project']['properties']['wasAssociatedWith']['items']['oneOf'][2]['properties']['identifier']['pattern'] = '^RRID:.*'
	this.schema = schema;
	},
	async fetchOwners(identifier: string) {
	const { data } = await dandiRest.owners(identifier);
	this.owners = data;
	},
	},
	});

[yarikoptic]

Thank you, nice digging! I think this was a missing piece. I will adjust #2381 accordingly

[candleindark]

Good point. With that in mind, instead of grabbing both id_pattern, regex pattern for the prefix of identifiers and instance_name, I think we can just have the user input the instance_name for dandischema and share it with dandi-archive. instance_name will be used as the prefix of the identifier and as part of a regex pattern internally, but to the user there is no semantic of the input being a regex pattern. Doing so will make things simpler for the user, eliminate the logic requires to verify that the instance_name conforms to id_pattern, and consolidate information in one place. We can also apply the same principle to the datacite_doi_id_pattern config field.

Overall I think it is sound, but I would prefer to make it so "by default" it would work for local installations. So instead of suggested

instance_name: str = Field(default=r"[A-Z]+", pattern=r"^[A-Z]+$")
"""Name of the DANDI instance"""

datacite_doi_id: str = Field(default=r"\d{4,}", pattern=r"^\d{4,}$")

what if we use

instance_name: str = Field(default=r"DANDI-ADHOC", pattern=r"^[A-Z][-A-Z]*$")
"""Name of the DANDI instance"""

datacite_doi_id: str = Field(default=r"", pattern=r"^(\d{4,}|)$")

thus making it we

• always have some instance name (DANDI-ADHOC) which we reserve for the default value if not vendorized

  • it would still match pattern/requirement to be non empty and capital
  • TODO: check if identifiers.org could have an identifier with - in the name. If "yes" - good", if "not" - let's change above to "DANDIADHOC" or alike and adjust pattern

• by default there would be no datacite_doi_id (empty). This should trigger total disablement of interactions with datacite (attn @asmacdo)

WDYT?

With my best guess of your intent, I have the following redefinition.

dandischema.conf.py

# This file defines the configuration for the DANDI schema

from __future__ import annotations

from typing import Annotated, Optional

from pydantic import StringConstraints
from pydantic_settings import BaseSettings, SettingsConfigDict


class Config(BaseSettings):
    """
    Configuration for the DANDI schema

    Note
    ----
        Since this class is subclass of `pydantic.BaseSettings`, each field of an
        instance of this class can be populated from an environment variable of the
        same name prefixed with the prefix defined in `model_config` with the name
        of the environment variable interpreted **case-insensitively**.
        For details, see https://docs.pydantic.dev/latest/concepts/pydantic_settings/
    """

    model_config = SettingsConfigDict(env_prefix="dandi_")

    instance_name: Annotated[str, StringConstraints(pattern=r"^[A-Z][-A-Z]*$")] = (
        "DANDI-ADHOC"
    )
    """Name of the DANDI instance"""

    datacite_doi_id: Optional[
        Annotated[str, StringConstraints(pattern=r"^\d{4,}$")]
    ] = None
    """
    The registrant code of the DOI prefix at DataCite

    The number sequence that follows "10." within the DOI prefix as documented
    at https://support.datacite.org/docs/prefixes.
    """

    @property
    def id_pattern(self) -> str:
        """Regex pattern for the prefix of identifiers"""
        if "instance_name" in self.model_fields_set:
            return self.instance_name

        # If the instance name is not set,
        #   we use a pattern for unvendored DANDI instances
        return r"[A-Z][-A-Z]*"

    @property
    def datacite_doi_id_pattern(self) -> str:
        """The registrant code pattern of the DOI prefix at DataCite"""
        if self.datacite_doi_id is not None:
            return self.datacite_doi_id

        # If the DataCite ID, registrant code, is not provided,
        #   we use a pattern for unvendored DANDI instances
        return r"\d{4,}"


CONFIG = Config()

Using this Config model has the following additional benefits.

1. Clear distinctions between
   1. instance_name and id_pattern
   2. datacite_doi_id and datacite_doi_id_pattern
2. id_pattern and datacite_doi_id_pattern default back to unvendored patterns when instance_name and datacite_doi_id are not provided respectively.
3. When datacite_doi_id is not provided, it has a unambiguous value of None, and datacite_doi_id_pattern assumes the unvendored pattern.

I will adjust dandi/dandi-schema#294 to use this model. Let me know if you'd like to make any changes though.

Note
id_pattern now defaults to r[A-Z][-A-Z]* instead of r[A-Z]+.

[yarikoptic]

quick followup: overall premise is good, but above looks "complicated" and duplicates regexes (and duplication is evil, already e.g. raising the question why we have id_pattern regex anchored in one case but not the other). May be just define some consts above to reuse here or would not work for StringConstraints?

[candleindark]

quick followup: overall premise is good, but above looks "complicated" and duplicates regexes (and duplication is evil, already e.g. raising the question why we have id_pattern regex anchored in one case but not the other). May be just define some consts above to reuse here or would not work for StringConstraints?

1. I just introduced some constants to reduce the complexity: https://github.com/dandi/dandi-schema/blob/5f056b985d9d6c53`3604f81f798ef94e2d626fc1/dandischema/conf.py#L10-L11. Thanks for the suggestion.
2. The patterns in instance_name and datacite_doi_id need the anchors because Pydantic doesn't do full matches of strings against the targeted patterns. It uses re.search. id_pattern and datacite_doi_id_pattern don't have anchors because they are used as part of a bigger patterns. (See how they are used in models.py at https://github.com/dandi/dandi-schema/blob/5f056b985d9d6c533604f81f798ef94e2d626fc1/dandischema/models.py#L47-L48.)
3. The complexity results from the fact that we needed a distinction between instance_name and id_pattern and a distinction between datacite_doi_id and datacite_doi_id_pattern. (The value of datacite_doi_id can't be used as the DataCite DOI ID pattern when it is not provided by the user).
4. The annotated pattern is a bit verbose, but it actually make the intend more clear. Not using the annotated pattern, I would have to write

    datacite_doi_id: Optional[str] = Field(None, pattern=r"^\d{4,}$")

instead

    datacite_doi_id: Optional[
        Annotated[str, StringConstraints(pattern=r"^\d{4,}$")]
    ] = None

The annotated pattern clearly indicates that the pattern metadata is to be applied to str instead of Optional[str]. I actually have seen Pydantic interprets some annotation incorrectly when the annotated pattern is not used.

[candleindark]

Most of the posts in the issues are outdated, e.g., the approach implemented by #2396 is abandoned at this point. Can we start another issue to reorient ourselves about what needs to be done on the dandi-archive side?

[yarikoptic]

yes, sounds good, please make it happen

[candleindark]

yes, sounds good, please make it happen

After re-reading the top post again. I think the top post is fine after I updated the link to the design document. Now, I think it is better to keep this issue but just to ignore the posts between this one and the top post since they are outdated.

With that said, I think we need to reopen #2430 since, according to the top post, it will be used by the web frontend as well.