pipt

Inversion (estimation, data assimilation)

ES

Bases: EnKF

Ensemble Smoother (ES).

Assimilates all observations simultaneously in a single update, rather than sequentially in time as the filter does. It is :class:EnKF specialised to one data group, and shares its analysis step; only the iteration budget and the misfit bookkeeping differ.

A single conditioning step is cheap but can over-correct when the model is strongly non-linear. :class:ESMDA addresses this by spreading the same update over several inflated steps.

Parameters:

Name	Type	Description	Default
keys_da	dict	Parsed dataassim configuration. Besides the keys every scheme reads -- data, datavar, obsname, truedataindex -- the ones this scheme acts on are listed under Notes.	required
keys_en	dict	Parsed ensemble configuration: ensemble size ne, the state variable names, and the prior_<name> blocks describing each.	required
sim	object	Forward simulator instance, e.g. simulator.opm.flow.	required
analysis	(approx, full, subspace)	Analysis flavour, i.e. how the ensemble-approximated sensitivity is inverted. Defaults to the analysis key in keys_da, falling back to 'approx'. The flavours differ in cost and in how they handle a rank-deficient ensemble; they solve the same update equation.	'approx'

Attributes:

Name	Type	Description
ensemble	AssimilationEnsemble	Collaborator holding the state realisations, observed data and simulator. Its state is exposed as properties on the scheme, so scheme.enX and scheme.keys_da read straight through.
analysis	AnalysisBase	The bound analysis object. Note the constructor takes analysis as a name and this attribute holds the resulting object, the way Model(optimizer="adam").optimizer is an optimizer instance.
analysis_name	str	The flavour name that was resolved, e.g. 'approx'.
iteration	int	Accepted iterations completed so far.
data_misfit, prior_data_misfit	float	Current and initial mean data misfit.

Notes

assimindex is flattened to a single group at construction, so the ordering that matters for :class:EnKF has no effect here.

Because there is only one step, the full flavour coincides with approx -- the prior-increment term they differ over is only reached when iterating -- so :attr:EnKF.COMPATIBLE_ANALYSES, inherited unchanged here, points "full" at the cheaper approx analysis.

Examples:

>>> result = ES.assimilate(keys_da, keys_en, flow(keys_sim))
>>> result.nit
1

References

Evensen, Data Assimilation: The Ensemble Kalman Filter evensen2009a.

See Also

EnKF : Sequential form of the same update. ESMDA : Spreads the conditioning over several inflated steps.

__init__(keys_da, keys_en, sim, analysis=None, ensemble=None)

Build the ensemble from the config (or take the one given) and bind the analysis.

See the class docstring for the parameters.

check_convergence()

ES takes a single all-data-at-once step; nothing stops early.

score_and_commit()

Calculate the "convergence" of the method. Important to

ESMDA

Bases: AssimilationScheme

Ensemble Smoother with Multiple Data Assimilation (ES-MDA).

An iterative ensemble smoother that assimilates all data repeatedly over a fixed number of steps, inflating the data-error covariance at each one so that the repeated conditioning does not over-fit. With inflation factors :math:\alpha_i satisfying :math:\sum_i 1/\alpha_i = 1, each step applies

.. math::

m \leftarrow m + C_{md} (C_{dd} + \alpha_i C_d)^{-1} (d_{obs} - g(m))

with the observations re-perturbed as :math:d_{obs} = d_{true} + \sqrt{\alpha_i} C_d^{1/2} Z.

The schedule is fixed rather than convergence-driven, so a run normally ends by exhausting its steps and reports success=False. That is the expected outcome, not a failure.

Parameters:

Name	Type	Description	Default
keys_da	dict	Parsed dataassim configuration. Besides the keys every scheme reads -- data, datavar, obsname, truedataindex -- the ones this scheme acts on are listed under Notes.	required
keys_en	dict	Parsed ensemble configuration: ensemble size ne, the state variable names, and the prior_<name> blocks describing each.	required
sim	object	Forward simulator instance, e.g. simulator.opm.flow.	required
analysis	(approx, full, subspace)	Analysis flavour, i.e. how the ensemble-approximated sensitivity is inverted. Defaults to the analysis key in keys_da, falling back to 'approx'. The flavours differ in cost and in how they handle a rank-deficient ensemble; they solve the same update equation.	'approx'

Attributes:

Name	Type	Description
ensemble	AssimilationEnsemble	Collaborator holding the state realisations, observed data and simulator. Its state is exposed as properties on the scheme, so scheme.enX and scheme.keys_da read straight through.
analysis	AnalysisBase	The bound analysis object. Note the constructor takes analysis as a name and this attribute holds the resulting object, the way Model(optimizer="adam").optimizer is an optimizer instance.
analysis_name	str	The flavour name that was resolved, e.g. 'approx'.
iteration	int	Accepted iterations completed so far.
data_misfit, prior_data_misfit	float	Current and initial mean data misfit.

Notes

Configured through the mda block of keys_da:

tot_assim_steps Number of assimilation steps, e.g. 3. inflation_param Inflation factors, one per step, e.g. [3, 3, 3]. Their reciprocals must sum to 1, which is asserted at construction. Defaults to tot_assim_steps repeated, which satisfies the constraint.

Examples:

>>> result = ESMDA.assimilate(keys_da, keys_en, flow(keys_sim))
>>> result.nit
3

References

Emerick and Reynolds, Ensemble smoother with multiple data assimilation emerick2013a.

See Also

ES : Single-step smoother; ES-MDA with one assimilation step. LMEnRML : Iterates to convergence instead of on a fixed schedule.

__init__(keys_da, keys_en, sim, analysis=None, ensemble=None)

Build the ensemble from the config (or take the one given) and bind the analysis.

See the class docstring for the parameters; ensemble is a ready-made collaborator to run on instead of building one.

calc_analysis()

Analysis step of ES-MDA. The analysis algorithm is similar to EnKF analysis, only difference is that the data covariance matrix is inflated with an inflation parameter alpha. The update is done as an iterative smoother where all data is assimilated at once.

Notes

ES-MDA is an iterative ensemble smoother with a predefined number of iterations, where the updates is done with the EnKF update equations but where the data covariance matrix have been inflated:

\[ \begin{align} d_{obs} &= d_{true} + \sqrt{\alpha}C_d^{1/2}Z \\ m &= m_{prior} + C_{md}(C_g + \alpha C_d)^{-1}(g(m) - d_{obs}) \end{align} \]

where \(d_{true}\) is the true observed data, \(\alpha\) is the inflation factor, \(C_d\) is the data covariance matrix, \(Z\) is a standard normal random variable, \(C_{md}\) and \(C_{g}\) are sample covariance matrices, \(m\) is the model parameter, and \(g(\)\) is the predicted data. Note that \(\alpha\) can have a different value in each assimilation step and must fulfill:

\[ \sum_{i=1}^{N_a} \frac{1}{\alpha} = 1 \]

where \(N_a\) being the total number of assimilation steps.

check_convergence()

ES-MDA runs its full schedule of inflated steps; nothing stops early.

log_columns(prior_run=False)

ES-MDA reports the inflation factor for the step just taken.

score(pred_data=None)

Data misfit against the un-inflated perturbed observations.

enObs is redrawn each step with the covariance inflated by alpha[iteration], so scoring against it would compare every iteration to a different yardstick. enObs_conv is the copy taken before any inflation, which is what makes the misfit trajectory comparable across the schedule.

score_and_commit()

Score the forecast that followed the analysis, then commit the step.

Was the second half of check_convergence: ES-MDA never actually tested for convergence there, it recomputed the misfit, logged the iteration and promoted enX_temp. Under the new contract the convergence question lives in :meth:check_convergence and this keeps the bookkeeping.

Returns:

Type	Description
dict	The why_stop record, also stored on self.why_stop.

update_step()

Run one ES-MDA assimilation step.

Computes the inflated analysis, forecasts the trial state, then scores the resulting misfit and promotes the state. Scoring after the forecast is what lets outlier replacement, which runs in between, feed into the number the scheme sees.

Returns:

Type	Description
bool	Always True. ES-MDA takes a fixed number of inflated steps and never rejects one. The success flag it logs compares the misfit against the previous iteration and is a reporting signal only -- returning it here would make the base class discard accepted steps.

EnKF

Bases: AssimilationScheme

Ensemble Kalman Filter (EnKF).

Assimilates data sequentially, updating the state once per group of observations in the order given by assimindex. Each update applies the Kalman equations with the covariances approximated from the ensemble:

.. math::

m \leftarrow m + C_{md} (C_{dd} + C_d)^{-1} (d_{obs} - g(m))

There is no damping and no rejection: every step is accepted, and the run ends once the data groups are exhausted.

Parameters:

Name	Type	Description	Default
keys_da	dict	Parsed dataassim configuration. Besides the keys every scheme reads -- data, datavar, obsname, truedataindex -- the ones this scheme acts on are listed under Notes.	required
keys_en	dict	Parsed ensemble configuration: ensemble size ne, the state variable names, and the prior_<name> blocks describing each.	required
sim	object	Forward simulator instance, e.g. simulator.opm.flow.	required
analysis	(approx, full, subspace)	Analysis flavour, i.e. how the ensemble-approximated sensitivity is inverted. Defaults to the analysis key in keys_da, falling back to 'approx'. The flavours differ in cost and in how they handle a rank-deficient ensemble; they solve the same update equation.	'approx'

Attributes:

Name	Type	Description
ensemble	AssimilationEnsemble	Collaborator holding the state realisations, observed data and simulator. Its state is exposed as properties on the scheme, so scheme.enX and scheme.keys_da read straight through.
analysis	AnalysisBase	The bound analysis object. Note the constructor takes analysis as a name and this attribute holds the resulting object, the way Model(optimizer="adam").optimizer is an optimizer instance.
analysis_name	str	The flavour name that was resolved, e.g. 'approx'.
iteration	int	Accepted iterations completed so far.
data_misfit, prior_data_misfit	float	Current and initial mean data misfit.

Notes

assimindex determines the grouping and ordering of the sequential updates. If all data are to be assimilated in a single step, use :class:ES, which is this scheme specialised to one group.

energy sets the fraction of singular values retained in the truncated SVD (default 0.98); values above 1 are read as percentages.

Every data group is assimilated exactly once, so the prior-increment term that distinguishes full from approx is never reached: "full" is pointed at the same class as "approx" in :attr:COMPATIBLE_ANALYSES. :class:ES inherits this.

Examples:

>>> result = EnKF.assimilate(keys_da, keys_en, flow(keys_sim))

References

Evensen, Data Assimilation: The Ensemble Kalman Filter evensen2009a.

See Also

ES : All-data-at-once form of the same update.

__init__(keys_da, keys_en, sim, analysis=None, ensemble=None)

Build the ensemble from the config and bind the analysis.

See the class docstring for the parameters.

calc_analysis()

Calculate the analysis step of the EnKF procedure. The updating is done using the Kalman filter equations, using svd for numerical stability. Localization is available.

check_convergence()

The EnKF runs its full sweep of data groups; nothing stops early.

score_and_commit()

Calculate the "convergence" of the method. Important to

update_step()

Run one EnKF step: analysis, forecast, then score and commit.

Returns:

Type	Description
bool	Always True. The EnKF applies one update per data group and has no rejection path.

GNEnRML

Bases: IterativeEnRML

Gauss-Newton Ensemble Randomized Maximum Likelihood (GN-EnRML).

Solves the same randomized maximum likelihood problem as :class:LMEnRML, but takes undamped Gauss-Newton steps scaled by a step length :math:\gamma \in (0, 1] rather than inflating the Hessian:

.. math::

m \leftarrow m + \gamma \, C_{md} (C_d + C_{dd})^{-1}
(d_{obs} - g(m))

Steps are accepted or rejected on the mean data misfit as in LM-EnRML. On acceptance :math:\gamma is relaxed towards gamma_max; on rejection it is divided by gamma_factor and the step re-solved, in the same within-:meth:update_step loop LM-EnRML uses for :math:\lambda.

Parameters:

Name	Type	Description	Default
keys_da	dict	Parsed dataassim configuration. Besides the keys every scheme reads -- data, datavar, obsname, truedataindex -- the ones this scheme acts on are listed under Notes.	required
keys_en	dict	Parsed ensemble configuration: ensemble size ne, the state variable names, and the prior_<name> blocks describing each.	required
sim	object	Forward simulator instance, e.g. simulator.opm.flow.	required
analysis	(approx, full, subspace)	Analysis flavour, i.e. how the ensemble-approximated sensitivity is inverted. Defaults to the analysis key in keys_da, falling back to 'approx'. The flavours differ in cost and in how they handle a rank-deficient ensemble; they solve the same update equation.	'approx'

Attributes:

Name	Type	Description
ensemble	AssimilationEnsemble	Collaborator holding the state realisations, observed data and simulator. Its state is exposed as properties on the scheme, so scheme.enX and scheme.keys_da read straight through.
analysis	AnalysisBase	The bound analysis object. Note the constructor takes analysis as a name and this attribute holds the resulting object, the way Model(optimizer="adam").optimizer is an optimizer instance.
analysis_name	str	The flavour name that was resolved, e.g. 'approx'.
iteration	int	Accepted iterations completed so far.
data_misfit, prior_data_misfit	float	Current and initial mean data misfit.

Notes

Configured through the iteration block of keys_da:

max_iter Maximum accepted iterations. gamma Initial step length (default 0.2). gamma_max Value the step length relaxes towards on success (default 0.5). gamma_factor Divisor applied to the step length on rejection (default 2.5). max_inner_iter Step-length attempts one iteration may make before the run gives up (default 10). There is no gamma_min, so this is what bounds it. data_misfit_tol Relative misfit change treated as converged (default 0.01).

The margis flavour is backed by margIS_update, ported from an older layout. It returns a matrix-form ensemble transform step (AnalysisResult(W_step=...), starting from W = I) rather than the weight step most other flavours use; propose_state reconstructs the state for either. Run against real data it produces a large, sensible misfit reduction, but is still one run on one case with no committed reference pinning it -- see its module docstring (:mod:pipt.update_schemes.analysis.margis) for what was fixed in the port and what remains a modelling choice rather than a bug.

Examples:

>>> result = GNEnRML.assimilate(keys_da, keys_en, flow(keys_sim))

References

Chen and Oliver chen2013; see also Raanes, Stordal and Evensen, Revising the stochastic iterative ensemble smoother raanes2019, and Evensen et al. evensen2019.

See Also

IterativeEnRML : The loop, scoring and bookkeeping both schemes share. LMEnRML : Levenberg-Marquardt form, damped via the Hessian.

log_columns(prior_run=False)

GN-EnRML reports the step length the logged iteration took.

LMEnRML

Bases: IterativeEnRML

Levenberg-Marquardt Ensemble Randomized Maximum Likelihood (LM-EnRML).

An iterative ensemble smoother that solves the randomized maximum likelihood problem by repeated linearisation, with a Levenberg-Marquardt damping parameter :math:\lambda controlling the step size. The damped update inflates the Hessian approximation:

.. math::

m \leftarrow m + C_{md} \big((1 + \lambda) C_d + C_{dd}\big)^{-1}
(d_{obs} - g(m))

Unlike ES-MDA, steps are accepted or rejected. A step that increases the mean data misfit is discarded, :math:\lambda is multiplied by lambda_factor and the step re-solved from the same state; one that decreases it is kept and :math:\lambda reduced. That retry loop lives inside :meth:update_step, so one iteration is one call however many attempts it takes -- the shape popt's optimizers have. The run stops when the relative misfit change falls below data_misfit_tol, when :math:\lambda reaches lambda_max, when a single iteration exhausts max_inner_iter attempts, or on max_iter.

Parameters:

Name	Type	Description	Default
keys_da	dict	Parsed dataassim configuration. Besides the keys every scheme reads -- data, datavar, obsname, truedataindex -- the ones this scheme acts on are listed under Notes.	required
keys_en	dict	Parsed ensemble configuration: ensemble size ne, the state variable names, and the prior_<name> blocks describing each.	required
sim	object	Forward simulator instance, e.g. simulator.opm.flow.	required
analysis	(approx, full, subspace)	Analysis flavour, i.e. how the ensemble-approximated sensitivity is inverted. Defaults to the analysis key in keys_da, falling back to 'approx'. The flavours differ in cost and in how they handle a rank-deficient ensemble; they solve the same update equation.	'approx'

Attributes:

Name	Type	Description
ensemble	AssimilationEnsemble	Collaborator holding the state realisations, observed data and simulator. Its state is exposed as properties on the scheme, so scheme.enX and scheme.keys_da read straight through.
analysis	AnalysisBase	The bound analysis object. Note the constructor takes analysis as a name and this attribute holds the resulting object, the way Model(optimizer="adam").optimizer is an optimizer instance.
analysis_name	str	The flavour name that was resolved, e.g. 'approx'.
iteration	int	Accepted iterations completed so far.
data_misfit, prior_data_misfit	float	Current and initial mean data misfit.

Notes

Configured through the iteration block of keys_da:

max_iter Maximum accepted iterations. lambda Initial damping parameter (default 100). 'auto' derives it from the prior data misfit. lambda_factor Factor by which damping grows on rejection and shrinks on acceptance (default 5). Held as lam_factor -- not gamma, which is GN-EnRML's step length, a different quantity entirely. lambda_max, lambda_min Bounds on the damping parameter. max_inner_iter Damping attempts one iteration may make before the run gives up (default 10). lambda_max normally stops it first. data_misfit_tol Relative misfit change treated as converged (default 0.01).

Examples:

>>> result = LMEnRML.assimilate(keys_da, keys_en, flow(keys_sim))
>>> result.message
'Maximum number of iterations reached'

success distinguishes the two ways a run can end: True when a convergence criterion fired, False when max_iter was reached first. Both are ordinary outcomes -- check prior_data_misfit against data_misfit to judge whether the run achieved anything.

References

Chen and Oliver, Levenberg-Marquardt forms of the iterative ensemble smoother for efficient history matching and uncertainty quantification chen2013.

See Also

IterativeEnRML : The loop, scoring and bookkeeping both schemes share. GNEnRML : Gauss-Newton form, damped by a step length instead. ESMDA : Fixed schedule rather than convergence-driven iteration.

log_columns(prior_run=False)

LM-EnRML reports the damping the logged iteration ran with.

score(pred_data=None)

Data misfit, sizing lambda='auto' the first time there is one.

:math:\lambda_0 = \Phi_{prior} / 2 N_d is defined against the prior misfit, so it cannot be settled in __init__. The first score of a run is the prior's, which makes this the earliest point it can be resolved -- and everything downstream needs a number: the prior row reports λ, and the prior QA/QC pass computes with it.

available_schemes()

Return the registered (scheme, analysis) combinations, sorted.

build_scheme(scheme, da_input, en_input, sim, analysis=None)

Construct any registered scheme by name.

Parameters:

Name	Type	Description	Default
scheme	str	Algorithm name, e.g. "esmda".	required
da_input	dict	Parsed data-assimilation config.	required
en_input	dict	Parsed ensemble config.	required
sim	object	Forward simulator instance.	required
analysis	str	Analysis flavour. Defaults to the config's analysis key, so that this agrees with :func:pipt.pipt_init.init_da, falling back to "approx" if the config does not say. Pass it to override the config.	None

Returns:

Type	Description
object	The instantiated scheme.

get_scheme(scheme, analysis)

Look up the constructor for a (scheme, analysis) combination.

Returns:

Type	Description
callable	Either the class directly (for a :data:SPECIAL_SCHEMES entry) or the algorithm class with analysis pre-bound via :func:functools.partial. Either way, call it as result(da_input, en_input, sim).

Raises:

Type	Description
KeyError	If the combination is not registered. The message distinguishes an unknown scheme from a known scheme with an unsupported analysis flavour, and lists the valid options in both cases.

register_scheme(scheme, analysis, cls, *, overwrite=False)

Add a scheme to the registry.

Parameters:

Name	Type	Description	Default
scheme	str	Scheme name, as it appears in the config's scheme key.	required
analysis	str	Analysis flavour, as it appears in the config's analysis key.	required
cls	type	Class implementing the combination.	required
overwrite	bool	Allow replacing an existing entry. Defaults to False so that two packages silently claiming the same key is an error rather than a load-order lottery.	False