This is the release note of v4.5.0.

Highlights

GPSampler for constrained multi-objective optimization

GPSampler is now able to handle multiple objective and constraints simultaneously using the newly introduced constrained LogEHVI acquisition function.

The figures below show the difference between GPSampler (LogEHVI, unconstrained) vs GPSampler (constrained LogEHVI, new feature). The 3-dimensional version of the C2DTLZ2 benchmark problem we used is a problem where some areas of the Pareto front of the original DTLZ2 problem are made infeasible by constraints. Therefore, even if constraints are not taken into account, it is possible to obtain the Pareto front. Experimental results show that both LogEHVI and constrained LogEHVI can approximate the Pareto front, but the latter has significantly fewer infeasible solutions, demonstrating its efficiency.

Optuna v4.4 (LogEHVI)	Optuna v4.5 (Constrained LogEHVI)

Significant speedup of TPESampler

TPESampler is significantly (about 5x as listed in the table below) faster! It enables a larger number of trials in each study. The speedup was achieved through a series of enhancements in constant factors.

The following table shows the speed comparison of TPESampler between v4.4.0 and v4.5.0. The experiments were conducted using multivariate=True on a search space with 3 continuous parameters and 3 numerical discrete parameters. Each row shows the runtime for each number of objectives and each column shows each number of trials to be evaluated. Each runtime is shown along with the standard error over 3 random seeds. The numbers in parentheses represent the speedup factor in comparison to v4.4.0. For example, (5.1x) means the runtime of v4.5.0 is 5.1 times faster than that of v4.4.0.

n_objectives/n_trials	500	1000	1500	2000
1	1.4 $\pm$ 0.03 (5.1x)	3.9 $\pm$ 0.07 (5.3x)	7.3 $\pm$ 0.09 (5.4x)	11.9 $\pm$ 0.10 (5.4x)
2	1.8 $\pm$ 0.01 (4.7x)	4.7 $\pm$ 0.02 (4.8x)	8.7 $\pm$ 0.03 (4.8x)	13.9 $\pm$ 0.04 (4.9x)
3	2.0 $\pm$ 0.01 (4.2x)	5.4 $\pm$ 0.03 (4.4x)	10.0 $\pm$ 0.03 (4.6x)	15.9 $\pm$ 0.03 (4.7x)
4	4.2 $\pm$ 0.11 (3.2x)	12.1 $\pm$ 0.14 (3.9x)	20.9 $\pm$ 0.23 (4.2x)	31.3 $\pm$ 0.05 (4.4x)
5	12.1 $\pm$ 0.59 (4.7x)	30.8 $\pm$ 0.16 (5.8x)	50.7 $\pm$ 0.46 (6.5x)	72.8 $\pm$ 1.13 (7.1x)

Significant speedup of plot_hypervolume_history

plot_hypervolume_history is essential to assess the performance of multi-objective optimization, but it was unbearably slow when a large number of trials are evaluated on a many-objective (The number of objectives > 3) problem. v4.5.0 addressed this issue by incrementally updating the hypervolume instead of calculating each hypervolume from scratch.

The following figure shows the elapsed times of hypervolume history plot in Optuna v4.4.0 and v4.5.0 using a four-objective problem. The x-axis represents the number of trials and the y-axis represents the elapsed times for each setup. The blue and red lines are the results of v4.4.0 and v4.5.0, respectively.

CmaEsSampler now supports 1D search space

Up until Optuna v4.4, CmaEsSampler could not handle one-dimensional space and fell back to random search. Optuna v4.5 now allows the CMA-ES algorithm to be used for one-dimensional space.

The optunahub library is available on conda-forge

Now, you can install the optunahub library via conda-forge as follows.

:::sh
conda install conda-forge::optunahub

New Features

• Add ConstrainedLogEHVI (#6198)
• Add support for constrained multi-objective optimization in GPSampler (#6224)
• Support 1D Search Spaces in CmaEsSampler (#6228)

Enhancements

• Move optuna._lightgbm_tuner module (https://github.com/optuna/optuna-integration/pull/233, thanks @milkcoffeen!)
• Fix numerical issue warning on qehvi_candidates_func (https://github.com/optuna/optuna-integration/pull/242, thanks @LukeGT!)
• Calculate hypervolume in HSSP using sum of contributions (#6130)
• Use hypervolume difference as upperbound of contribs in HSSP (#6131)
• Refactor tell_with_warning to avoid unnecessary get_trial call (#6133)
• Print fully qualified name of experimental function by default (#6162, thanks @ktns!)
• Include scipy-stubs in the type-check dependencies (#6174, thanks @jorenham!)
• Warn when GPSampler falls back to RandomSampler (#6179, thanks @sisird864!)
• Handle slowdown of GPSampler due to L-BFGS in SciPy v1.15 (#6191)
• Use the Newton method instead of bisect in ndtri_exp (#6194)
• Speed up erf for TPESampler (#6200)
• Avoid duplications in _log_gauss_mass evaluations (#6202)
• Remove unnecessary NumPy usage (#6215)
• Use subset comparator to judge if trials are included in search space (#6218)
• Speed up log pdf in _BatchedTruncNormDistributions by vectorization (#6220)
• Speed up WFG by skipping is_pareto_front and using simple Python loops (#6223)
• Vectorize ndtri_exp (#6229)
• Speed up plot_hypervolume_history (#6232)
• Speed up HSSP 4D+ by using a decremental approach (#6234)
• Use lru_cache to skip HSSP (#6240, thanks @fusawa-yugo!)
• Add hypervolume computation for a zero size array (#6245)

Bug Fixes

• Fix: Resolve PG17 incompatibility for ENUMS in CASE statements (#6099, thanks @vcovo!)
• Fix ill-combination of journal and gRPC (#6175)
• Fix a bug in constrained GPSampler (#6181)
• Fix TPESampler with multivariate and constant_liar (#6189)

Installation

• Remove version constraint for torch with Python 3.13 (#6233)

Documentation

• Add missing spaces in error message about inconsistent intermediate values (https://github.com/optuna/optuna-integration/pull/239, thanks @Greesb!)
• Improve parallelization document (#6123)
• Add FAQ for case sensitivity problem with MySQL (#6127, thanks @fusawa-yugo!)
• Add an FAQ entry about specifying optimization parameters (#6157)
• Update link to survey (#6169)
• Add introduction of OptunaHub in docs (#6171, thanks @fusawa-yugo!)
• Add GPSampler as a sampler that supports constraints (#6176, thanks @1kastner!)
• Remove optuna-fast-fanova references from documentation (#6178)
• Fix stale docs in GP-related modules (#6184)
• Embed link to OptunaHub in documentation (#6192, thanks @fusawa-yugo!)
• Update README.md (#6222, thanks @muhammadibrahim313!)
• Add a note about unrelated changes in PR (#6226)
• Document the default evaluator in Optuna Dashboard (#6238)

Examples

• Add Spark example using ask-and-tell interface (https://github.com/optuna/optuna-examples/pull/328, thanks @dhyeyinf!)

Tests

• Fix test_log_completed_trial_skip_storage_access (#6208)

Code Fixes

• Clean up GP-related docs (#6125)
• Refactor return style [#6136] (#6151, thanks @unKnownNG!)
• Refactor KernelParamsTensor towards cleaner GP-related modules (#6152)
• Rename KernelParamsTensor to GPRegressor (#6153)
• Refactor returns in v3.0.0.d.py (#6154, thanks @dross20!)
• Refactor acquisition function minimally (#6166)
• Implement Type-Checking for optuna/_imports.py (#6167, thanks @AdrianStrymer!)
• Fix type checking in optuna.artifacts._download.py (#6177, thanks @dross20!)
• Integrate is_categorical to search space (#6182)
• Fix type checking in optuna.artifacts._list_artifact_meta.py (#6187, thanks @dross20!)
• Introduce the independent sampling warning template (#6188)
• Use warning template for independent sampling in GPSampler (#6195)
• Refactor SearchSpace in GP (#6197)
• Refactor _truncnorm (#6201)
• Use TYPE_CHECKING in optuna/_gp/acqf.py to avoid circular imports (#6204, thanks @CarvedCoder!)
• Use TYPE_CHECKING in optuna/_gp/optim_mixed.py to avoid circular imports (#6205, thanks @Subodh-12!)
• Flip the sign of constraints in GPSampler (#6213)
• Implement NSGA-III using BaseGASampler (#6219)
• Replace torch.newaxis with None for old PyTorch (#6237)

Continuous Integration

• Fix CI (https://github.com/optuna/optuna-integration/pull/238)
• Add sklearn version constraint (https://github.com/optuna/optuna-integration/pull/243)
• Fix README for blackdoc==0.3.10 (#6150)
• Fix CI (#6161)
• Add pytest-xdist to speed up the CI (#6170)
• Delete test_get_timeline_plot_with_killed_running_trials (#6210)
• Fix fragile test test_experimental (#6211)
• Mark fragile test as xfail (#6217)
• Add unit tests for constrained multi-objective GPSampler (#6235)
• Fix CI (#6246)

Other

• Update the example list in the README (https://github.com/optuna/optuna-integration/pull/234, thanks @ParagEkbote!)
• Bump up version number to 4.5.0.dev (https://github.com/optuna/optuna-integration/pull/237)
• Bump up the version number to 4.5.0 (https://github.com/optuna/optuna-integration/pull/244)
• Bump up version number to v4.5.0.dev (#6149)
• Update News section in README (#6159)
• Bump up version to v4.5.0 (#6251)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@1kastner, @AdrianStrymer, @CarvedCoder, @Greesb, @HideakiImamura, @LukeGT, @ParagEkbote, @Subodh-12, @c-bata, @contramundum53, @dhyeyinf, @dross20, @fusawa-yugo, @gen740, @hvy, @jorenham, @kAIto47802, @ktns, @milkcoffeen, @muhammadibrahim313, @nabenabe0928, @not522, @nzw0301, @sawa3030, @sisird864, @toshihikoyanase, @unKnownNG, @vcovo, @y0z