Introducing Dandiset DOIs #2350

asmacdo · 2025-04-22T18:25:04Z

POC Implementation for Design document for the Zenodo like DOI per dandiset #2012

Example of Dandiset DOI (note -- no version) : 10.80507/dandi.000004

Remaining TODOs:

May be later as part of this depending on what comes in first

Rebase schema PR on Isaac's changes

dandiapi/api/doi.py

asmacdo · 2025-05-14T18:09:22Z

Getting closer to completion now, but there are a few things I have not yet been able to verify:

Deletion of dandisets: I should have superuser but I dont see anything on /admin so I cant actually delete the dandisets to verify DOI delete/hide behavior.
Findable/Registered DOIs: when DANDI_DOI_PUBLISH is true, we create findable DOIs which might behave differently. I'll need to patch the DOIs manually to prevent collisions in our test datacite api, and modify dandischema to allow that. A PITA, so I'm doing other stuff first.
Unembargo workflow: I ran into 3 issues:

CODE_OF_CONDUCT.md

asmacdo · 2025-05-20T15:33:31Z

dandiapi/api/datacite.py

+        self.api_url = settings.DANDI_DOI_API_URL
+        self.api_user = settings.DANDI_DOI_API_USER
+        self.api_password = settings.DANDI_DOI_API_PASSWORD
+        self.api_prefix = settings.DANDI_DOI_API_PREFIX


Previously, we would use settings.DANDI_DOI_API_PREFIX or '10.80507'

If api prefix is not set, DOI API operations should be prevented by is_configured() so I think this is appropriate

dandiapi/api/services/dandiset/__init__.py

dandiapi/api/services/publish/__init__.py

dandiapi/api/services/dandiset/__init__.py

asmacdo · 2025-05-22T20:59:06Z

dandiapi/api/services/dandiset/__init__.py

todo should still be a celery task, but we should make sure it passes or roll back transaction

I don't agree that it must pass, or we should roll back the transaction. For starters, we would have to move all of this code into the celery task to ensure that. More practically, why should the outcome of the datacite API call influence this? Handling failure conditions for interacting with the datacite API should be done anyway. It wouldn't help anyone to prevent the deletion of a dandiset due to the datacite API being down (for example). We would just need to ensure that it is eventually deleted.

For example, one way to approach this is to, upon dandiset deletion, place the DOI of the draft version into a table called DeletableDOI (or something). Then, a scheduled celery task could pull from values in this table, call out to datacite to delete the DOI, and then in the same transaction, delete that row (or mark is as "done") so it's not picked up on the next scheduled task.

This would ensure that the DOI eventually gets deleted, even if there is a transient error during interaction with the datacite API. The same procedure could be implemented for DOI creation as well.

My main concern was not to orphan the doi, but this seems like a reasonable approach

yarikoptic

just posting an exception we observerd while trying locally. Ideally should be more graceful I think

dandiapi/api/doi.py

asmacdo · 2025-06-03T15:41:23Z

+1 we can handle that more gracefully. This exception occurred because the Dandiset was created by the superuser without a name. So FWIW even if we relax how to_datacite sets that value it will still be an invalid DOI payload.

yarikoptic · 2025-07-30T19:31:58Z

@waxlamp @jjnesbitt This item is on our weekly meeting notes "beyond 30 minutes of Roni" so we rarely get to it. Could someone give the initial review? this "feature" was being in the kitchen for way too long with the design doc PR (#2012) and now this grew big and potentially growing conflicts.

jjnesbitt · 2025-07-30T19:33:11Z

@waxlamp @jjnesbitt This item is on our weekly meeting notes "beyond 30 minutes of Roni" so we rarely get to it. Could someone give the initial review? this "feature" was being in the kitchen for way too long with the design doc PR (#2012) and now this grew big and potentially growing conflicts.

Yes, I am in the process of reviewing it.

jjnesbitt · 2025-07-30T17:21:07Z

dandiapi/api/datacite.py

+                # Retry with PUT if DOI already exists
+                update_url = f'{self.api_url}/{doi}'


If the DOI already exists, what does making a PUT request do? Update the metadata?

Yeah exactly

jjnesbitt · 2025-07-30T20:05:34Z

dandiapi/api/services/dandiset/__init__.py

I don't agree that it must pass, or we should roll back the transaction. For starters, we would have to move all of this code into the celery task to ensure that. More practically, why should the outcome of the datacite API call influence this? Handling failure conditions for interacting with the datacite API should be done anyway. It wouldn't help anyone to prevent the deletion of a dandiset due to the datacite API being down (for example). We would just need to ensure that it is eventually deleted.

For example, one way to approach this is to, upon dandiset deletion, place the DOI of the draft version into a table called DeletableDOI (or something). Then, a scheduled celery task could pull from values in this table, call out to datacite to delete the DOI, and then in the same transaction, delete that row (or mark is as "done") so it's not picked up on the next scheduled task.

This would ensure that the DOI eventually gets deleted, even if there is a transient error during interaction with the datacite API. The same procedure could be implemented for DOI creation as well.

jjnesbitt · 2025-07-30T20:12:55Z

dandiapi/api/services/dandiset/__init__.py

+        draft_version = dandiset.versions.filter(version='draft').first()
+        if draft_version and draft_version.doi is not None:


I have a couple of suggestions. First, we know a draft version will always exist on a dandiset, as it is created in the same transaction as the dandiset. So we don't need to check that it exists here.

Second, I actually think we should prevent deleting the dandiset until the DOI is generated. Imagine someone creates and then immediately deletes a dandiset, at the same time the celery task to create the DOI is running. This code block would be skipped, and the dandiset would still be deleted, but the DOI could have been created during that time. In that case, we would have "orphaned DOIs" to worry about.

I think it's acceptable to block deletion on draft DOI creation, and raise an appropriate error message from the request.

jjnesbitt · 2025-07-30T20:25:17Z

dandiapi/api/views/version.py

+                # For unpublished dandisets, update or create the draft DOI
+                # to keep it in sync with the latest metadata
+                if not locked_version.dandiset.embargoed:
+                    transaction.on_commit(
+                        lambda: update_draft_version_doi_task.delay(locked_version.id)
+                    )
+                else:
+                    logger.debug(
+                        'Skipping DOI update for embargoed Dandiset %s.',
+                        locked_version.dandiset.identifier,
+                    )


If we're going to be keeping doi metadata up to date anytime a draft version is updated, then I'm even more in favor of what I proposed in my other comment (a table for keeping track of this). Doing it in this fashion is just a disaster waiting to happen in regards to synchronization of metadata.

yarikoptic · 2025-08-27T18:32:25Z

Hi @jjnesbitt , should we wait on you to implement that extra table (then when do you think you would need time) or should we attempt at doing that?

jjnesbitt · 2025-09-03T14:41:59Z

Hi @jjnesbitt , should we wait on you to implement that extra table (then when do you think you would need time) or should we attempt at doing that?

I can take this on, as I've already begun the implementation.

- Dandiset DOI will redirect to the DLP - Example: 10.80507/dandi.000004 - Dandiset DOI is stored in the doi field of the draft version - Dandiset DOI metadata (on Datacite) will match the draft version until first publication - Once a Dandiset is published, the Dandiset DOI metadata will match the latest publication See the design document for more details: dandi#2012

The datacite_client_unsafe bypasses the safety check, but can still be used safely when mocking requests.

…tead

yarikoptic · 2025-09-05T20:39:02Z

@jjnesbitt how do you feel about this PR/work overall -- could/should we get it closer/over the finish line in the near future?

jjnesbitt · 2025-10-01T23:01:17Z

Apologies for the delay, I've gotten back to working on this and have made significant progress. For ease of development I've made a separate branch based off this one, and will open that PR once it's ready (hopefully soon).

asmacdo commented Apr 22, 2025

View reviewed changes

dandiapi/api/doi.py Outdated Show resolved Hide resolved

yarikoptic reviewed May 14, 2025

View reviewed changes

CODE_OF_CONDUCT.md Show resolved Hide resolved

asmacdo commented May 20, 2025

View reviewed changes

yarikoptic reviewed May 21, 2025

View reviewed changes

dandiapi/api/services/dandiset/__init__.py Outdated Show resolved Hide resolved

yarikoptic reviewed May 21, 2025

View reviewed changes

dandiapi/api/services/publish/__init__.py Outdated Show resolved Hide resolved

yarikoptic reviewed May 21, 2025

View reviewed changes

dandiapi/api/services/dandiset/__init__.py Outdated Show resolved Hide resolved

asmacdo force-pushed the enh-dandiset-dois branch 5 times, most recently from 5316e66 to 73de829 Compare May 22, 2025 20:29

asmacdo commented May 22, 2025

View reviewed changes

asmacdo force-pushed the enh-dandiset-dois branch from 138e463 to c1a0215 Compare May 22, 2025 22:47

yarikoptic reviewed Jun 3, 2025

View reviewed changes

dandiapi/api/doi.py Outdated Show resolved Hide resolved

yarikoptic mentioned this pull request Jun 4, 2025

Design document for the Zenodo like DOI per dandiset #2012

Merged

4 tasks

asmacdo force-pushed the enh-dandiset-dois branch 3 times, most recently from a5b9d24 to 3fd6081 Compare June 16, 2025 20:14

kabilar requested review from waxlamp, jjnesbitt and mvandenburgh July 14, 2025 18:58

jjnesbitt requested changes Jul 30, 2025

View reviewed changes

jjnesbitt force-pushed the enh-dandiset-dois branch 2 times, most recently from 03bee6c to 65e8306 Compare September 3, 2025 16:40

jjnesbitt force-pushed the enh-dandiset-dois branch 2 times, most recently from 95ef02b to 39dd8a8 Compare September 4, 2025 20:22

asmacdo added 17 commits September 4, 2025 16:24

Move DOI handling for publish to celery task

101471a

Add safety: disable calls to Datacite during tests

9e5879d

The datacite_client_unsafe bypasses the safety check, but can still be used safely when mocking requests.

fixup: safety feature caught one test using datacite, should mock ins…

95931ba

…tead

move doi handling for creation to celery task

f7e00f9

fixup tests for moved _create_dandiset_draft_doi

ab2e82e

move doi handling for update to celery task

c645c75

Cleanup unnecessary changes

8b27967

Add Dandiset DOI to vue

2acf10a

Prevent dandiset deletion if doi delete fails

a25ad0b

remove unnecessary compatability layer

5d8bea2

fixup linting

bb76239

Remove duplication

67d78e5

add delete management command

00af13b

test and fix integration with dandi schema changes

97df5c0

remove dandiset delete mgmnt command, not ready, will add later

3d41c8c

Add vue test for DOI on draft dandiset

988bb7b

jjnesbitt force-pushed the enh-dandiset-dois branch from 39dd8a8 to 988bb7b Compare September 4, 2025 20:25

		# Retry with PUT if DOI already exists
		update_url = f'{self.api_url}/{doi}'

		draft_version = dandiset.versions.filter(version='draft').first()
		if draft_version and draft_version.doi is not None:

Introducing Dandiset DOIs #2350

Are you sure you want to change the base?

Introducing Dandiset DOIs #2350

Uh oh!

Conversation

asmacdo commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

asmacdo commented May 14, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asmacdo commented Jun 3, 2025

Uh oh!

yarikoptic commented Jul 30, 2025

Uh oh!

jjnesbitt commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yarikoptic commented Aug 27, 2025

Uh oh!

jjnesbitt commented Sep 3, 2025

Uh oh!

yarikoptic commented Sep 5, 2025

Uh oh!

jjnesbitt commented Oct 1, 2025

Uh oh!

Uh oh!

asmacdo commented Apr 22, 2025 •

edited

Loading