978 Migrate Local Users, Datasets, Folders and Files

[tcnichol]

To test:

If you have a local running V1 instance, you can use that. And take a look at notes below to properly set up V1 instance.
↓↓↓↓
Otherwise you can use the demo instance if possible.

• Create an .env file and place it under the migration folder
• This .env should look like below

CLOWDER_V1=https://clowder.ncsa.illinois.edu/clowder
ADMIN_KEY_V1={v1 api key}

CLOWDER_V2=http://0.0.0.0:8000
ADMIN_KEY_V2={v2 api key}

Adjust the v1 users line for testing

---

The first part of user migration. Here we will migrate users from a clowder v1 instance into a clowder v2 instance. This works for local users, datasets, folders, and files.

I have refactored this so that there is a 'migrate_user' method. This will probably be a useful pattern to follow since we might want to handle CILogon users differently. We also might want to make migration something a user can initiate from another instance, where there might be options they need to select. This might become important once we get into collection hierarchies, how to handle spaces and sharing, metadata, and then things like licenses.

A later pull request will address collection hierarchies, metadata, CILogon accounts, and other features.

To test this, you will want to run a clowder v1 instance. Since both need to run a different version of MongoDB, change this in the .yml file for the dependencies in v1.

  # database to hold metadata (required)
  mongo:
    image: mongo:3.6
    restart: unless-stopped
    networks:
      - clowder
    command: mongod --port 27018
    ports:
      - '27018:27018'
    volumes:
      - mongo:/data/db

also change this in application.conf

mongodbURI = "mongodb://127.0.0.1:27018/clowder"

You will also need to do add this line in application.conf so that the emails will be printed in terminal

smtp.mock=true

It can be added anywhere.

To run v1, use

docker-compose up mongo -d

And then run clowder v1 using IntelliJ.
If you add entries to the v1 instance, try migrating them.

Also, you will need an admin user from the v1 instance and an api key to be placed in a .env file.

This can also be tested with another instance. The migration is handled using the API so you can use an existing instance and try to migrate.

TODO:

• metadata definition @longshuicy
• metadata @longshuicy
• collections @tcnichol
• spaces @todd
• visualizations
• license @ddey2