Available Containerized Benchmarks

A list of all available benchmarks

For more details, we refer to the code of each benchmark in hpobench.benchmark.<family>.

benchmarks.ml

Family	Name	Type	Note
pybnn(x)	BNNOn*	raw	There are 4 benchmark in total (ToyFunction, BostonHousing, ProteinStructure, YearPrediction)
lr_benchmark	LRBenchmark	raw	Works on OpenML Dataset IDs
nn_benchmark	NNBenchmark	raw	Works on OpenML Dataset IDs
rf_benchmark	RandomForestBenchmark	raw	Works on OpenML Dataset IDs
svm_benchmark	SVMBenchmark	raw	Works on OpenML Dataset IDs
tabular_benchmark(x)	TabularBenchmark	tabular	Tabular benchmarks for the benchmarks above ('lr', 'svm', 'xgb', 'rf', 'nn')
histgb_benchmark	HistGBBenchmark	raw	Works on OpenML Dataset IDs
xgboost_old	XGBoost*Benchmark	raw	Works with OpenML task ids. extended contains Additional Parameter `Booster
svm_benchmark_old	SupportVectorMachine	raw	Works with OpenML task ids.

benchmarks.rl

Family	Name	Type	Note
cartpole	CartpoleFull	raw	Not deterministic.
cartpole(x)	CartpoleReduced	raw	Not deterministic. Uses a smaller searchspace than full
learna_benchmark	LeaRNA	Not deterministic.
learna_benchmark	MetaLearna	Not deterministic.

benchmarks.nas

Family	Name	Type	Note
tabular_benchmarks(+)	*Benchmark	tabular	There are 4 benchmarks (SlizeLocalization, ProteinStructure, NavalPropulsion, ParkinsonsTelemonitoring)
tabular_benchmarks(+,x)	*BenchmarkOriginal	tabular	Same as above, but with a higher lowest fidelity of 3
nasbench_101(+)	NASCifar10*Benchmark	tabular	There are 3 benchmark in total (A, B, C)
nasbench_201(+)	*NasBench201Benchmark	tabular	There are 3 benchmarks in total (Cifar10Valid, Cifar100, ImageNet). Note: We assume that there are 3 seeds per configuration (777,888,999). For configurations with less seeds we re-use of the existing ones (See #138).
nasbench_201(+,x)	*NasBench201BenchmarkOriginal	tabular	Same as above, but with a higher lowest fidelity of 12
nasbench_1shot1(+)	NASBench1shot1SearchSpace*Benchmark	tabular	There are 3 benchmarks in total (1,2,3)

benchmarks.surrogates

Family	Name	Type	Note
paramnet_benchmark	ParamNet*OnStepsBenchmark	paramnet	surrogate
paramnet_benchmark	ParamNetReduced*OnStepsBenchmark	paramnet	surrogate
paramnet_benchmark	ParamNet*OnTimeBenchmark	paramnet	surrogate
paramnet_benchmark(x)	ParamNetReduced*OnTimeBenchmark	paramnet	surrogate
svm_benchmark	SurrogateSVMBenchmark	surrogate_svm	surrogate

benchmarks.od

Family	Name	Type	Note
od_ae	ODAutoencoder	raw	Works on 15 datasets (annthyroid, arrhythmia, breastw, cardio, ionosphere, mammography, musk, optdigits, pendigits, pima, satellite, satimage-2, thyroid, vowels, wbc)
od_kde	ODKernelDensityEstimation	raw	Works on 15 datasets (see above). No fidelities available
od_ocsvm	ODOneClassSupportVectorMachine	raw	Works on 15 datasets (see above). No fidelities available

Note:

• (x) benchmarks that have been used in the HPOBench paper, other benchmarks are considered experimental
• (+) Tabular benchmarks might take a few minutes to load.

Note: All containers are uploaded here