Experiments for the non Archival Track of AutoMLConf 2025.
Python 3.9 as well as anaconda are required to run the experiments. Tested on macOS and Linux.
Create and activate a version 3.9 python environment:
conda create -n automatic_stopping python=3.9
conda activate automatic_stopping
Install swig, the local, customized smac-version, and the requirements:
pip install swig
pip install ../SMAC3
pip install -r requirements.txt
You might want to pre-download the datasets from the OpenML CC-18 benchmark to speed up the experiments:
python 02_openml_cc18/openml_cc18_download_datasets_by_taskid.py
If you do not specify a download-location beforehand, they will be cached in your home-directory.
Our experiments rely on the library. They therefore require either using a mysql or sqlite database. The process of using PyExperimenter is described in its documentation.
To run the experiments, first fill the pyexperimetner table using 09_python_scripts/openml_cc18_smac_fill_table.py. This will fill the table with all tasks and their respective configurations.
Then, run the experiments using 09_python_scripts/openml_cc18_smac_evaluate.py. This will run the experiments and fill the results table in the database.
To plot the results, use the plotting scripts in the 02_openml_cc18/plotting directory. The scripts will read the results from the database and create plots.