-
Notifications
You must be signed in to change notification settings - Fork 0
quickstart
This is an example on how to setup the docker version of the platform with a sample challenge.
Clone this repository on your server git clone [email protected]:zerospeech/vocolab.git /location/vocolab
You need to have docker & docker-compose installed.
Go into the repository and edit the docker-compose.yml file, to configure base settings for the platform.
In the api service replace the following env variables with your own :
VC_API_BASE_URL: "https://api.vocolab.com"
VC_maintainers: "CoML Team, INRIA, ENS, EHESS, CNRS"
VC_admin_email: "[email protected]"
VC_MAIL_USERNAME: "[email protected]"
VC_MAIL_PASSWORD: "XXXXXXX"
VC_MAIL_FROM: "[email protected]"
VC_MAIL_FROM_NAME: "Vocolab Challenges"
VC_MAIL_PORT: 25
VC_MAIL_SERVER: "mail.vocolab.com"
VC_MAIL_TLS: "True"
VC_MAIL_SSL: "False"
API_BASE_URL defines the entry point for the api you need to replace it with your own domain. admin_email defines the email of the platform administrator MAIL**_******* defines SMTP setting which are usefull for user authentification.
If you do not have an smtp server you can use a gmail account as smpt more info here.
This is the minimal necessairy setup for the platform to be able to work correctly, there is a more detailed description on all the configuration options available in the Configuration page.
Once configuration is done you can build your docker container using the command docker-compose up -d
This will build all the images & run the containers in the background.
To verify everything was setup correctly you can navigate using your browser to http://localhost:8000 which will show some basic infomation on the API :
{"app":"VocoLab Challenge API","version":"v0.5","maintainers":"CoML Team, INRIA, ENS, EHESS, CNRS","contact":"[email protected]","installation_datetime":"2022-06-14T15:00:29.691039"}
You can also navigate to http://localhost:8000/docs to preview the API documentation.
Test that the CLI is correctly setup by running docker-compose exec api voco help, this should show you a list of commands available if everything was setup correctly.
The samples folder is mounted on /samples in your api container this is usefull if you want to access external files.
The first thing you will need to do once the platform is up and running is to create a challenge so that users can submit to it.
Inside the samples folder, create a challenge.json file containing the following :
[{
"id": 1
"label": "test-challenge",
"start_date": "2022-06-30",
"end_date": null,
"url": "https://vocolab.com/challenge/test",
"active": false,
"evaluator": null
}]
This allows creating a challenge named test-challenge that starts at the 30th of June 2022 without an end
date.
The URL referrers to the page that contains information about the challenge if you have not set it up, you can leave this field empty for now.
As with all examples you can replace all the fields with your own values.
Run the following command to create your challenge:
`docker-compose exec voco challenges:add -f /samples/challenge.json`
You should be able to see your challenge by running :
`docker-compose exec voco challenges -a`
You might notice that we set our challenge to non-active, this means it will not accept submissions at this point.
The next step will be to create a leaderboard so that we can show submission results.
Create a leaderboard.json file in the samples directory :
[{
"label": "test-leaderboard",
"challenge_id": 1,
"path_to": "test-leaderboard.json",
"entry_file": "entry.json",
"external_entries": "/app-data/archive/test-archives",
"archived": false,
"static_files": false
}]
Once this file is done we run the command :
`docker-compose exec api voco leaderboards:create -f /samples/leaderboard.json`
This will create the leaderboard entry.
You can see your leaderboards by running
docker-compose exec api voco leaderboards
label: the name of the leaderboard
challenge_id: the linked challenge
path_to: the name of the file containing the leaderboard.
entry_file: the filename containing each entry inside the submissions (this should be were the evaluator outputs the submissions score)
external_entries: allows to add precomputed entries to the leaderboard (baseline, topline etc...), entries need to be in independant files either in .yaml or .json format.
archived: allows to set a leaderboard as non dynamic, no new submissions can be added.
static_files: this allows to extract some static files from evaluation (such as generated audio/image etc).
In this example we use the default docker setup in which the eval worker is in the same server as the api so no networking/storage sync needs to happen. In most cases a different server might be needed for evaluation as they might be costly so you can see how to setup remote evaluation in the Evaluation Section.
First, we need to create an evaluation script. An evaluation script will receive as argument the path to the submission entry that needs to be evaluated.
For more information on submission directory structure see Submissions Section
The evaluator DIR is set up to the containers/evaluators directory which is mounted on API and evaluation worker.
As an example we will use the test_eval.py which generates a random scores.json file, you can replace it with your own file.
Once your script is ready, we need to add it to the index.yml file at the root of the directory, so it can be discoverable.
evaluators:
test:
executor: bash
script_path: /app-data/evaluators/test_eval.sh
executor_arguments:
- "-x" # run verbose mode
test2:
executor: python
script_path: /app-data/evaluators/test_eval.py
executor_arguments:
- "-OO" # run with optimizations
executor: the type of executor to use (python, bash, etc...)
In the default docker version only python & bash are available, you need to either alter the base worker.Dockerfile to add your own dependencies or perform a native installation of your eval worker.
script_path: the absolute path of your script. executor_arguments: a list of optional arguments to pass to your executor.
Once the script and the index are done we need to register the evaluators, we can do that using the following command :
`docker-compose exec api voco evaluators:discover localhost`
You can list the evaluators registered with this command :
`docker-compose exec api voco evaluators`
Next, we can link our evaluator to our challenge so that all submissions use this script for evaluation :
`docker-compose exec api voco challenges:set 1 evaluator 2`
We can see that the evaluator has been correctly set up using the command :
`docker-compose exec api voco challenges -a`
And then proceed to activate your challenge :
`docker-compose exec api voco challenges:set 1 active true`
Now, as long as the challenge has not expired (reached end_date) it will accept submissions.
A challenge can accept submissions as long as its active and the current date is between start and end date parameters.
If we want to test our evaluation pipeline, we need first to create a user
A user can be created by going to http://localhost:8000/page/new-user And fill in your information, once the form is filled you should receive an email allowing you to validate your account.
You can list all users in the platform using the command :
docker-compose exec api voco users
Now that the user is created, you can upload a sample submission to your challenge using our upload script.
This script is an example client script you can adapt its function to suit your needs.
First create a python environment and install dependencies :
$ python3 -m venv myenv
$ source myenv/bin/activate
$ pip install pandas rich filesplit==3.0.2 pycryptodome requests
Once this is done, you can upload your submission.zip :
$ python3 scripts/upload.py submission.zip
This should upload your zip to the server and launch auto-evaluation.
$ docker-compose exec api voco submissions
Should show your submission as evaluating or completed if the process has completed. Once it is completed, you can run :
$ docker-compose exec voco leaderboards:show 1
And see the scores computed.
You can also visit http://localhost:8000/leaderboards/1/json to see the same leaderboard from the API.
Of course in a production environment you will not access your API from the localhost URL so you need to configure a reverse proxy to allow external users to connect to your API. See how to setup a reverse proxy in the Setup Section
Your users will access your backend via the API, so they will need some tools to help them. You will probably have a website describing your challenge and how to participate. A set of tools to interact with the backend will need to be provided in addition to VocoLab.
For description of a challenge the content of the website is mostly static, we find that the best tool to build such a website is to use static site generators, such as hugo, sphinx, jekyll or any other.
You can see our website used for the Zero Resource Challenge Series as an example. This site was created using hugo.
To create a dynamic leaderboard, our recommendation is to use the DataTables JavaScript library, which allows you to build dynamic tables from json objects. With a setup like this, you could connect your datatable to pull data directly from the API (https://api.example.com/leaderboard/1/json) and make your leaderboard auto-update when a new submission is added.
For submission upload, we have an example script (scripts/upload.py) that you can modify to suit your case. You could also add an upload section on your page, by using the API directly in JavaScript.