analysis_tools

Collection of tools that can be used in update/analysis schemes.

Only put tools here that are so general that they can be used by several update/analysis schemes. If some method is only applicable to the update scheme you are implementing, leave it in that class.

aug_obs_pred_data(obs_data, pred_data, assim_index, list_data)

Augment the observed and predicted data to an array at an assimilation step. The observed data will be an augemented vector and the predicted data will be an ensemble matrix.

Parameters:

Name	Type	Description	Default
obs_data	list	List of dictionaries containing observed data	required
pred_data	list	List of dictionaries where each entry of the list is the forward simulation results at an assimilation step. The dictionary has keys equal to the data type (given in OBSNAME).	required

Returns:

Name	Type	Description
obs	ndarray	Augmented vector of observed data
pred	ndarray	Ensemble matrix of predicted data

aug_state(state, list_state, cell_index=None)

Augment the state variables to an array.

Parameters:

Name	Type	Description	Default
state	dict	Dictionary of initial ensemble of (joint) state variables (static parameters and dynamic variables) to be assimilated.	required
list_state	list	Fixed list of keys in state dict.	required
cell_index	list of vector indexes to be extracted		None

Returns:

Name	Type	Description
aug	ndarray	Ensemble matrix of augmented state variables

calc_autocov(pert)

Calculate sample auto-covariance matrix.

Parameters:

Name	Type	Description	Default
pert	ndarray	Perturbation matrix (matrix of variables perturbed with their mean)	required

Returns:

Name	Type	Description
cov_auto	ndarray	Sample auto-covariance matrix

calc_objectivefun(pert_obs, pred_data, Cd)

Calculate the objective function.

Parameters:

Name	Type	Description	Default
pert_obs	array - like	NdxNe array containing perturbed observations.	required
pred_data	array - like	NdxNe array containing ensemble of predictions.	required
Cd	array - like	NdxNd array containing data covariance, or Ndx1 array containing data variance.	required

Returns:

Name	Type	Description
data_misfit	array - like	Nex1 array containing objective function values.

calc_scaling(enX, idX, prior_info)

Form the scaling to be used in svd related algoritms. Scaling consist of standard deviation for each STATICVAR It is important that this is formed in the same manner as the augmentet state vector is formed. Hence, with the same list of states.

Parameters:

Name	Type	Description	Default
enX	ndarray	State ensemble matrix, shape (nx, ne); only its row count per variable is used.	required
idX	dict	Row range (start, stop) of each state variable in enX, in the order the state was stacked.	required
prior_info	dict	Nested dictionary containing prior information	required

Returns:

Name	Type	Description
scaling	numpy array	scaling

gen_covdata(datavar, assim_index, list_data)

Generate the data covariance matrix at current assimilation step. Note here that the data covariance may be a diagonal matrix with only variance entries, or an empirical covariance matrix, or both if in combination. For diagonal data covariance we only store vector of variance values.

Parameters:

Name	Type	Description	Default
datavar	list	List of dictionaries containing variance for the observed data. The structure of this list is the same as for obs_data	required
assim_index	int	Current assimilation index	required
list_data	list	List of the data types	required

Returns:

Name	Type	Description
cd	ndarray	Data auto-covariance matrix

Notes

For empirical covariance generation, the datavar entry must be a 2D array, arranged as a standard ensemble matrix (N x Ns, where Ns is the number of samples).

get_list_data_types(obs_data, assim_index)

Extract the list of all and active data types

Parameters:

Name	Type	Description	Default
obs_data	dict	Observed data	required
assim_index	int	Current assimilation index	required

Returns:

Name	Type	Description
l_all	list	List of all data types
l_act	list	List of the data types that are active (that are not None)

get_outlier_index(pred, data, data_var=None, tresh=4.0)

Identify outlier ensemble members based on a normalized data-mismatch score.

For each ensemble member j, the mismatch is:

h_j = sum_i ((Y_ij - d_i) / sigma_i)^2

where sigma_i is the ensemble standard deviation (or provided variance) for observable i. Members whose score deviates more than tresh standard deviations from the mean are flagged as outliers.

Parameters:

Name	Type	Description	Default
pred	(array_like, shape(nd, ne))	Predicted data ensemble, one column per member.	required
data	(array_like, shape(nd))	Observed data, in the same row order.	required
data_var	array_like or None	Data variance, (nd,) or (nd, ne) for an empirical ensemble. If not provided, the ensemble variance of the predicted data is used.	None
tresh	float	Outlier threshold in numbers of standard deviations. Default is 4.	4.0

Returns:

Name	Type	Description
outlier_indices	ndarray	Indices of outlier ensemble members.
members	ndarray	Array of ensemble member indices, with outliers replaced by randomly selected non-outlier members.

limits(state, prior_info)

Check if any state variables overshoots the limits given by the prior info. If so, modify these values

Parameters:

Name	Type	Description	Default
state	dict	Dictionary containing the states	required
prior_info	dict	Dictionary containing prior information for all the states.	required

Returns:

Name	Type	Description
state	dict	Valid state

parallel_upd(list_state, prior_info, states_dict, X, local_mask_info, obs_data, pred_data, parallel, actnum=None, field_dim=None, act_data_list=None, scale_data=None, num_states=1, emp_d_cov=False)

Script to initialize and control a parallel update of the ensemble state following emerick2016a.

Parameters:

Name	Type	Description	Default
list_state	list	List of state names	required
prior_info	dict	INSERT DESCRIPTION	required
states_dict	dict	Dict. of state arrays	required
X	ndarray	INSERT DESCRIPTION	required
local_mask_info	dict	INSERT DESCRIPTION	required
obs_data	ndarray	Observed data	required
pred_data	ndarray	Predicted data	required
parallel	int	Number of parallel runs	required
actnum	ndarray	Active cells	None
field_dim	list	Number of grid cells in each direction	None
act_data_list	list	List of active data names	None
scale_data	ndarray	Scaling array for data	None
num_states	int	Number of states	1
emp_d_cov	bool	INSERT DESCRIPTION	False

Notes

Since the localization matrix is to large for evaluation, we instead calculate it row for row.

save_analysisdebug(ind_save, **kwargs)

Deprecated alias for :func:save_assimilation_result.

The files are not a debugging aid -- they are the per-iteration record of a run -- so both the function and what it writes were renamed.

save_assimilation_result(ind_save, **kwargs)

Save the requested variables for one assimilation iteration.

The PIPT counterpart to popt.misc_tools.optim_tools.save_optimize_results, which writes optimize_result_{i}.npz.

Parameters:

Name	Type	Description	Default
ind_save	int	Iteration index. 0 is the prior.	required
**kwargs	dict	Variables that will be saved to npz file	{}

Notes

Use kwargs here because the input will be a dictionary with names equal the variable names to store, and when this is passed to np.savez (kwargs) the variable will be stored with their original name.

store_ensemble_sim_information(saveinfo, member)

Here, we can either run a unique python script or do some other post-processing routines. The function should not return anything, but provide a method for storing revevant information. Input the current member for easy storage

truncSVD(matrix, r=None, energy=None, full_matrices=False)

Perform truncated SVD on input matrix.

Parameters:

Name	Type	Description	Default
matrix	(ndarray, shape(m, n))	Input matrix to perform SVD on.	required
r	int	Rank to truncate the SVD to. If None, energy must be specified.	None
energy	float	Fraction of the singular-value sum to retain, given either as a fraction in (0, 1] or as a percentage in (1, 100]. The smallest rank whose retained fraction reaches this value is used, so the requested amount is met rather than approached from below. Note this accumulates the singular values themselves, not their squares -- it is a fraction of the nuclear norm, not of the Frobenius energy. If None, r must be specified.	None
full_matrices	bool	Whether to compute full or reduced SVD. Default is False.	False

Returns:

Name	Type	Description
U	(ndarray, shape(m, r))	Left singular vectors.
S	(ndarray, shape(r))	Singular values.
VT	(ndarray, shape(r, n))	Right singular vectors transposed.

update_state(aug_state, state, list_state, cell_index=None)

Extract the separate state variables from an augmented state array. It is assumed that the augmented state array is made in aug_state, hence this is the reverse method of aug_state.

Parameters:

Name	Type	Description	Default
aug_state	ndarray	Augmented array of UPDATED state variables	required
state	dict	Dict. of state variables NOT updated.	required
list_state	list	List of state keys that have been updated	required
cell_index	list	List of indexes that gives the where the aug state should be placed	None

Returns:

Name	Type	Description
state	dict	Dict. of UPDATED state variables