optimization_methods

Optimizers: EnOpt, GenOpt, LineSearch, TrustRegion and SmcOpt, all built on OptimizerBase.

BoundTransformHandler

Transform states between the original parameter domain and the unit cube.

Notes

All bounds must be finite whenever bounds are supplied.

__init__(bounds=None, transform=False)

Initialize the BoundTransformHandler.

Parameters:

Name	Type	Description	Default
bounds	sequence of (lower, upper) pairs	Lower and upper bounds for each state variable.	None
transform	bool	If True, transform the optimization problem to the unit cube [0, 1]^n.	False

hess_from_unit_cube(hess)

Transform a Hessian from unit-cube coordinates.

hess_to_unit_cube(hess)

Transform a Hessian to unit-cube coordinates.

jac_from_unit_cube(jac)

Transform a gradient from unit-cube coordinates.

jac_to_unit_cube(jac)

Transform a gradient to unit-cube coordinates.

project_gradient(x, g, tol=1e-08)

Project a gradient to respect active bound constraints.

project_to_bounds(x)

Project a vector onto the feasible domain.

state_to_unit_cube(x)

Transform original coordinates to unit-cube coordinates.

unit_cube_to_state(u)

Transform unit-cube coordinates to original coordinates.

EnOpt

Bases: OptimizerBase

Ensemble-based optimization (EnOpt).

obj_func_values

Legacy alias for fk.

__init__(x0, fun, jac=None, hess=None, args=(), bounds=None, callback=None, **options)

Initialize an EnOpt optimizer instance.

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial control/state vector.	required
fun	callable	Objective function.	required
jac	callable	Ensemble gradient function.	None
hess	callable	Ensemble Hessian function.	None
args	tuple	The first tuple element is interpreted as the initial covariance.	()
bounds	sequence	Lower and upper bounds for each state variable.	None
callback	callable	Callback invoked after successful updates.	None
**options		EnOpt configuration, plus everything :class:OptimizerBase takes. - tol: Convergence tolerance for objective improvement (default: 1e-6). Also used as ftol when given. - step_size: Initial optimizer step size. Overrides alpha when provided. - alpha: Initial optimizer step size (default: 0.1). - alpha_cov: Covariance update scaling factor (default: 0.001). - beta: Momentum parameter used in the optimizer and optional Nesterov updates (default: 0.0). - nesterov: Whether to evaluate search quantities with Nesterov momentum (default: False). - alpha_maxiter: Maximum number of backtracking trials per iteration (default: 5). - resample: Number of covariance resampling attempts if no improvement is found (default: 0). - hessian: Whether to use the Hessian in the search direction computation (default: False). - normalize: Whether to normalize the gradient or Hessian-derived search quantities (default: True). - cov_factor: Covariance shrink factor applied during resampling (default: 0.5). - optimizer: Update rule name. Supported values are GD, Adam, AdaMax, and Steihaug (default: GD).	{}

log_columns()

The row of the iteration log: iteration, backtracking attempts, objective, step size, first covariance entry.

update_step()

Perform one EnOpt step with backtracking and optional resampling.

GenOpt

Bases: OptimizerBase

Generalized ensemble optimization with an adapting mutation distribution.

EnOpt draws its ensemble from a Gaussian whose covariance is fixed apart from an optional Hessian-driven update. GenOpt draws from the marginals of :class:~popt.ensembles.ensemble_generalized.GeneralizedEnsemble -- Beta, logistic, truncated Gaussian -- and moves the distribution itself along with the controls: theta (the marginal's shape) follows its own gradient, and the correlation matrix follows corr_adapt.

So each accepted step updates three things rather than one: the controls from jac, theta from jac_mut, and corr from corr_adapt -- which is either a :class:CMA instance, called with the ensemble the mutation gradient was built from, or any callable returning a matrix to descend along.

Examples:

ensemble = GeneralizedEnsemble(options, simulator, objective)
cma = CMA(ne=ensemble.num_samples, dim=x0.size, corr_update=True)
result = GenOpt.minimize(
    x0, ensemble.function,
    jac=ensemble.gradient, jac_mut=ensemble.mutation_gradient,
    args=(ensemble.get_theta(), ensemble.get_corr()),
    corr_adapt=cma, bounds=bounds,
)

__init__(x0, fun, jac=None, jac_mut=None, corr_adapt=None, args=(), bounds=None, callback=None, **options)

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial control vector.	required
fun	callable	Objective function.	required
jac	callable	Ensemble gradient, called as jac(x, theta, corr).	None
jac_mut	callable	Mutation gradient, called as jac_mut(x, theta, corr). For a :class:CMA corr_adapt it is called with return_ensembles=True and must then also return {'gaussian': ..., 'objective': ...}.	None
corr_adapt	CMA or callable	Correlation-matrix adaptation. A :class:CMA instance is called with the ensemble; any other callable is called with no arguments and its result is descended along with step size alpha_corr. None leaves the correlation fixed.	None
args	tuple	(theta, corr): the initial marginal parameter and correlation matrix.	()
bounds	sequence	(min, max) per control.	None
callback	callable	Invoked after each accepted step.	None
**options		GenOpt configuration, plus everything :class:OptimizerBase takes. tol: objective improvement required to accept a step (default: 1e-6). alpha: initial step size for the controls (default: 0.1). alpha_theta: step size for the marginal parameter (default: 0.1). alpha_corr: step size for the correlation, for a non-CMA corr_adapt (default: 0.1). beta: momentum (default: 0.0). nesterov: evaluate the gradients at the momentum-extrapolated point (default: False). alpha_maxiter: backtracking trials per iteration (default: 5). resample: resampling attempts when backtracking fails (default: 0). normalize: scale both gradients by their inf-norm (default: True). cov_factor: shrink factor applied to theta when resampling (default: 0.5). optimizer: GD or Adam (default: GD).	{}

log_columns()

Iteration, backtracking attempts, objective, step size, and the correlation's spread.

update_step()

One GenOpt step: controls by backtracking, then theta and the correlation.

LineSearch

Bases: OptimizerBase

Line-search optimizer compatible with OptimizerBase.

The class supports gradient descent, BFGS, and Newton-CG search directions, together with either Wolfe or backtracking line search. It can operate with bounds, optional state transformations, logging, result persistence, and restart checkpoints.

__init__(x0, fun, method='GD', jac=None, hess=None, args=(), bounds=None, callback=None, **options)

Initialize a line-search optimizer instance.

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial parameter vector.	required
fun	callable	Objective function.	required
method	(GD, BFGS, Newton - CG)	Search-direction method.	'GD'
jac	callable	Gradient function.	None
hess	callable	Hessian function, required by Newton-CG.	None
args	tuple	Extra positional arguments passed to the wrapped callables.	()
bounds	sequence	Lower and upper bounds for each state variable.	None
callback	callable	Callback invoked after successful updates.	None
**options		Line-search configuration, plus everything :class:OptimizerBase takes. - step_size: Initial step size (default: None, auto-scaled). - step_size_max: Maximum step size (default: 1e5). - step_size_adapt: Step size adaptation strategy (0: none, 1: function-based, 2: gradient-based). Default is 1 (function-based). - c1: Armijo condition constant (default: 1e-4). - c2: Curvature condition constant (default: 0.9). - rho: Step size reduction factor for backtracking (default: 0.5). - lsmaxiter: Maximum line search iterations (default: 10). - lsmethod: Line search method (0: backtracking, 1: Wolfe, default: 1). - normalize: Whether to normalize the search direction (default: False). - recompute_jac: Number of gradient recomputation attempts on line search failure (default: 0). - hess0_inv: Initial inverse-Hessian approximation for BFGS (default: identity).	{}

log_columns()

The row of the iteration log: iteration, objective, gradient infinity norm, step length taken.

update_step()

Perform one optimization step.

The method computes a search direction, performs a line search, and commits the new iterate on success. When enabled, it can recompute the gradient and retry if the line search fails.

OptimizerBase

Bases: OptimizerRestartMixin, ABC

The iteration every optimizer shares; a subclass supplies the step.

A subclass implements :meth:update_step, committing an improving point with :meth:_commit_step and returning a :class:StepReport, and names what its log row shows in :meth:log_columns. Everything else -- the starting evaluation, the callback, recording and saving the result, the log row, the function, state and projected-gradient convergence checks, restart checkpoints and the EPF outer loop -- happens here.

NAME = 'Optimizer'

Shown in the start-of-run banner.

__init__(x0, fun, jac=None, hess=None, args=(), bounds=None, callback=None, **options)

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial parameter vector.	required
fun	callable	Objective function.	required
jac	callable	Gradient function.	None
hess	callable	Hessian function.	None
args	tuple	Extra positional arguments passed to callables: fun, jac, hess.	()
bounds	sequence	Lower and upper bounds for each state variable.	None
callback	callable	Called with the optimizer after every accepted step.	None
**options		Optimizer configuration such as tolerances, logging, restart, and persistence options. - maxiter: Maximum number of iterations (default: 100) - ftol: Relative function tolerance for convergence (default: 1e-5) - xtol: Relative change in state for convergence (default: 1e-8) - gtol: Projected-gradient infinity-norm tolerance for convergence (default: 1e-5) - fun0, jac0, hess0: Initial objective, gradient and Hessian values to reuse instead of evaluating them - logit: Enable logging (default: True) - logger_name: Log file name (default: 'OPTIM.log') - restart: Enable restart from file (default: False) - restartsave: Save restart file after each iteration (default: False) - restart_file: Path for restart file (default: '{optimizer_name}_restart.pkl') - epf: Dictionary of EPF options (default: None) - r: Initial penalty factor - r_factor: Penalty factor update multiplier (default: 2) - tol_factor: Function tolerance update multiplier (default: 0.9) - conv_crit: EPF convergence criterion, compared against the mean penalty with the penalty factor divided out (default: 1e-5). The objective must write penalty into the epf dict it is handed. - transform: Enable [lb, ub] → [0, 1] transformation for optimization (default: False) - saveit: Save intermediate results after each iteration (default: False) - savefolder (or save_folder): Folder for those results (default: 'Iteration_Results')	{}

check_convergence()

Optimizer-specific criteria; by default the projected gradient against gtol.

Runs after the function and state checks. An optimizer with more criteria extends this; one without a gradient gets False.

check_epf_convergence()

Evaluate convergence of the outer EPF iteration.

The loop stops once the constraints are satisfied, measured as the mean of self.epf['penalty'] with the penalty factor r divided back out. The objective is responsible for writing penalty into the epf dict it is handed; without it there is nothing to converge on and this raises.

Returns:

Type	Description
bool	True when the EPF loop should terminate, otherwise False.

check_function_convergence()

Check convergence based on relative change in objective value.

check_state_convergence()

Check convergence based on the norm of the state update.

log_columns()

One row of the iteration log. Optimizers override to show their own quantities.

minimize(x0, fun, *args, **kwargs)

Construct the optimizer with these arguments, run it, and return its result.

The arguments are the constructor's, in the constructor's order; see the class for what each optimizer takes.

run_optimization()

Run this optimizer to completion.

Named for the job rather than the mechanism; the counterpart in pipt is AssimilationScheme.run_assimilation.

The loop handles restart restoration, the starting evaluation, optional EPF outer iterations, repeated calls to update_step(), and the shared convergence checks. When enabled, restart files are updated after successful iterations and after EPF penalty updates.

update_step()

Take one step from the current iterate.

Find a better point and make it current with :meth:_commit_step, which also keeps the previous iterate for the convergence checks; then return StepReport(True). The loop runs the callback, records and saves the result, logs a row and checks convergence -- none of that is the step's job. Return StepReport(False, why) when no acceptable step exists: the run stops and why is its message.

OptimizerRestartMixin

Bases: RestartMixin

Checkpoint/restart behaviour for optimizers.

The implementation is shared with PIPT via :class:ensemble.checkpoint.RestartMixin; this subclass exists so the optimizer-facing name stays stable.

SmcOpt

Bases: OptimizerBase

Sequential Monte-Carlo optimizer with resampling and backtracking.

obj_func_values

Legacy alias for fk.

__init__(x0, fun, sens=None, args=(), bounds=None, callback=None, **options)

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial state	required
fun	callable	objective function	required
sens	callable	Ensemble sensitivity function	None
args	tuple	Initial covariance tuple where args[0] is the covariance matrix used for sampling.	()
bounds	list	(min, max) pairs for each element in x. None is used to specify no bound.	None
callback	callable	Callback invoked after successful updates.	None
options	dict	SmcOpt configuration, plus everything :class:OptimizerBase takes (transform is forced off: SmcOpt works in physical coordinates). tol: convergence tolerance for the objective function (default 1e-6). Also used as ftol when given. alpha: weight between previous and new step (default 0.1) alpha_maxiter: maximum number of backtracking trials (default 5) resample: number indicating how many times resampling is tried if no improvement is found cov_factor: factor used to shrink the covariance for each resampling trial (default 0.5) inflation_factor: term used to weight down prior influence (default 1.0) survival_factor: fraction of surviving samples (clipped to [0.1, 1.0]) best_func: best objective value seen before this run (default: the initial objective) savefolder/save_folder: folder used when saveit is true (default './')	{}

log_columns()

The row of the iteration log: iteration, backtracking attempts, objective, best objective seen, step size.

update_step()

Perform one SMC update step with backtracking and optional resampling.

StepReport

What one call to :meth:OptimizerBase.update_step produced.

accepted says the optimizer committed a new iterate (through :meth:OptimizerBase._commit_step); the loop then does the bookkeeping every optimizer used to repeat. message is why it stopped when it did not, and becomes the result's message.

TrustRegion

Bases: OptimizerBase

Trust-region Optimizer.

The class supports exact Hessian trust-region subproblems (iterative or CG-Steihaug) and optional BFGS Hessian approximation via hess='BFGS'.

__init__(x0, fun, jac, hess, method='iterative', args=(), bounds=None, callback=None, **options)

Initialize a trust-region optimizer instance.

Parameters:

Name	Type	Description	Default
x0	ndarray	Initial parameter vector.	required
fun	callable	Objective function.	required
jac	callable	Gradient function.	required
hess	callable or {BFGS}	Hessian function, or 'BFGS' to use a quasi-Newton Hessian approximation.	required
method	(iterative, CG - Steihaug)	Trust-region subproblem solver.	'iterative'
args	tuple	Extra positional arguments passed to the wrapped callables.	()
bounds	sequence	Lower and upper bounds for each state variable.	None
callback	callable	Callback invoked after successful updates.	None
**options		Trust-region configuration, plus everything :class:OptimizerBase takes. - trust_radius: Initial trust-region radius (default: 1.0). - trust_radius_max: Maximum trust-region radius (default: 100 * trust_radius). - trust_radius_min: Minimum trust-region radius before termination (default: trust_radius / 1000). - trust_radius_cuts: Maximum number of radius reductions before rejecting a step (default: 4). - rho_tol: Minimum ratio between actual and predicted reduction for step acceptance (default: 1e-6). - eta1: Threshold for rejecting a step (default: 0.05). - eta2: Threshold for increasing the trust-region radius (default: 0.5). - gam1: Factor used to decrease the trust-region radius (default: 0.5). - gam2: Factor used to increase the trust-region radius when the boundary is hit (default: 1.5). - resample: Whether to recompute gradient and Hessian after rejected steps (default: False). - convergence_criteria: Optional callable for custom convergence checks.	{}

check_convergence()

The projected gradient, the trust-region radius, and any custom criterion.

log_columns()

The row of the iteration log: iteration, objective, trust radius, reduction ratio, whether the step hit the boundary.

update_step()

Perform one trust-region step with optional radius reductions.