analysis_tools

Collection of tools that can be used in update/analysis schemes.

Only put tools here that are so general that they can be used by several update/analysis schemes. If some method is only applicable to the update scheme you are implementing, leave it in that class.

aug_obs_pred_data(obs_data, pred_data, assim_index, list_data)

Augment the observed and predicted data to an array at an assimilation step. The observed data will be an augemented vector and the predicted data will be an ensemble matrix.

Parameters:

Name	Type	Description	Default
obs_data	list	List of dictionaries containing observed data	required
pred_data	list	List of dictionaries where each entry of the list is the forward simulation results at an assimilation step. The dictionary has keys equal to the data type (given in OBSNAME).	required

Returns:

Name	Type	Description
obs	ndarray	Augmented vector of observed data
pred	ndarray	Ensemble matrix of predicted data

aug_state(state, list_state, cell_index=None)

Augment the state variables to an array.

Parameters:

Name	Type	Description	Default
state	dict	Dictionary of initial ensemble of (joint) state variables (static parameters and dynamic variables) to be assimilated.	required
list_state	list	Fixed list of keys in state dict.	required
cell_index	list of vector indexes to be extracted		None

Returns:

Name	Type	Description
aug	ndarray	Ensemble matrix of augmented state variables

block_diag_cov(cov, list_state)

Block diagonalize a covariance matrix dictionary.

Parameters:

Name	Type	Description	Default
cov	dict	Dict. with cov. matrices	required
list_state	list	Fixed list of keys in state dict.	required

Returns:

Name	Type	Description
cov_out	ndarray	Block diag. matrix with prior covariance matrices for each state.

calc_autocov(pert)

Calculate sample auto-covariance matrix.

Parameters:

Name	Type	Description	Default
pert	ndarray	Perturbation matrix (matrix of variables perturbed with their mean)	required

Returns:

Name	Type	Description
cov_auto	ndarray	Sample auto-covariance matrix

calc_crosscov(pert1, pert2)

Calculate sample cross-covariance matrix.

Parameters:

Name	Type	Description	Default
pert1		Perturbation matrices (matrix of variables perturbed with their mean).	required
pert2		Perturbation matrices (matrix of variables perturbed with their mean).	required

Returns:

Name	Type	Description
cov_cross	ndarray	Sample cross-covariance matrix

calc_kalman_filter_eq(aug_state, kalman_gain, obs_data, pred_data)

Calculate the updated augment state using the Kalman filter equations

Parameters:

Name	Type	Description	Default
aug_state	ndarray	Augmented state variable (all the parameters defined in STATICVAR augmented in one array)	required
kalman_gain	ndarray	Kalman gain	required
obs_data	ndarray	Augmented observed data vector (all OBSNAME augmented in one array)	required
pred_data	ndarray	Augmented predicted data vector (all OBSNAME augmented in one array)	required

Returns:

Name	Type	Description
aug_state_upd	ndarray	Updated augmented state variable using the Kalman filter equations

calc_kalmangain(cov_cross, cov_auto, cov_data, opt=None)

Calculate the Kalman gain

Parameters:

Name	Type	Description	Default
cov_cross	ndarray	Cross-covariance matrix between state and predicted data	required
cov_auto	ndarray	Auto-covariance matrix of predicted data	required
cov_data	ndarray	Variance on observed data (diagonal matrix)	required
opt	str	Which method should we use to calculate Kalman gain 'lu': LU decomposition (default) 'chol': Cholesky decomposition	None

Returns:

Name	Type	Description
kalman_gain	ndarray	Kalman gain

Notes

In the following Kalman gain is \(K\), cross-covariance is \(C_{mg}\), predicted data auto-covariance is \(C_{g}\), and data covariance is \(C_{d}\).

With 'lu' option, we solve the transposed linear system: $$ K^T = (C_{g} + C_{d})^{-T}C_{mg}T $$

With 'chol' option we use Cholesky on auto-covariance matrix, $$ L L^T = (C_{g} + C_{d})^T $$ and solve linear system with the square-root matrix from Cholesky: $$ L^T Y = C_{mg}^T\ LK = Y $$

calc_objectivefun(pert_obs, pred_data, Cd)

Calculate the objective function.

Parameters:

Name	Type	Description	Default
pert_obs	array - like	NdxNe array containing perturbed observations.	required
pred_data	array - like	NdxNe array containing ensemble of predictions.	required
Cd	array - like	NdxNd array containing data covariance, or Ndx1 array containing data variance.	required

Returns:

Name	Type	Description
data_misfit	array - like	Nex1 array containing objective function values.

calc_scaling(state, list_state, prior_info)

Form the scaling to be used in svd related algoritms. Scaling consist of standard deviation for each STATICVAR It is important that this is formed in the same manner as the augmentet state vector is formed. Hence, with the same list of states.

Parameters:

Name	Type	Description	Default
state	dict	Dictionary containing the state	required
list_state	list	List of states for augmenting	required
prior_info	dict	Nested dictionary containing prior information	required

Returns:

Name	Type	Description
scaling	numpy array	scaling

calc_subspace_kalmangain(cov_cross, data_pert, cov_data, energy)

Compute the Kalman gain in a efficient subspace determined by how much energy (i.e. percentage of singluar values) to retain. For more info regarding the implementation, see Chapter 14 in evensen2009a.

Parameters cov_cross : ndarray Cross-covariance matrix between state and predicted data data_pert : ndarray Predicted data - mean of predicted data cov_data : ndarray Variance on observed data (diagonal matrix)

Returns:

Name	Type	Description
k_g	ndarray	Subspace Kalman gain

compute_x(pert_preddata, cov_data, keys_da, alfa=None)

INSERT DESCRIPTION

Parameters:

Name	Type	Description	Default
pert_preddata	ndarray	Perturbed predicted data	required
cov_data	ndarray	Data covariance matrix	required
keys_da	dict	Dictionary with every input in DATAASSIM	required
alfa	None	INSERT DESCRIPTION	None

Returns:

Name	Type	Description
X	ndarray	INSERT DESCRIPTION

extract_tot_empirical_cov(data_var, assim_index, list_data, ne)

Extract realizations of noise from data_var (if imported), or generate realizations if only variance is specified (assume uncorrelated)

Parameters:

Name	Type	Description	Default
data_var	list	List of dictionaries containing the varianse as read from the input	required
assim_index	int	Index of the assimilation	required
list_data	list	List of data types	required
ne	int	Ensemble size	required

Returns:

Name	Type	Description
E	ndarray	Sorted (according to assim_index and list_data) matrix of data realization noise.

gen_covdata(datavar, assim_index, list_data)

Generate the data covariance matrix at current assimilation step. Note here that the data covariance may be a diagonal matrix with only variance entries, or an empirical covariance matrix, or both if in combination. For diagonal data covariance we only store vector of variance values.

Parameters:

Name	Type	Description	Default
datavar	list	List of dictionaries containing variance for the observed data. The structure of this list is the same as for obs_data	required
assim_index	int	Current assimilation index	required
list_data	list	List of the data types	required

Returns:

Name	Type	Description
cd	ndarray	Data auto-covariance matrix

Notes

For empirical covariance generation, the datavar entry must be a 2D array, arranged as a standard ensemble matrix (N x Ns, where Ns is the number of samples).

get_list_data_types(obs_data, assim_index)

Extract the list of all and active data types

Parameters:

Name	Type	Description	Default
obs_data	dict	Observed data	required
assim_index	int	Current assimilation index	required

Returns:

Name	Type	Description
l_all	list	List of all data types
l_act	list	List of the data types that are active (that are not None)

init_local_analysis(init, state)

Initialize local analysis.

Initialize the local analysis by reading the input variables, defining the parameter classes and search ranges. Build the map of data/parameter positions.

Args

init : dictionary containing the parsed information form the input file. state : list of states that will be updated

Returns:

Name	Type	Description
local	dictionary of initialized values.

limits(state, prior_info)

Check if any state variables overshoots the limits given by the prior info. If so, modify these values

Parameters:

Name	Type	Description	Default
state	dict	Dictionary containing the states	required
prior_info	dict	Dictionary containing prior information for all the states.	required

Returns:

Name	Type	Description
state	dict	Valid state

parallel_upd(list_state, prior_info, states_dict, X, local_mask_info, obs_data, pred_data, parallel, actnum=None, field_dim=None, act_data_list=None, scale_data=None, num_states=1, emp_d_cov=False)

Script to initialize and control a parallel update of the ensemble state following emerick2016a.

Parameters:

Name	Type	Description	Default
list_state	list	List of state names	required
prior_info	dict	INSERT DESCRIPTION	required
states_dict	dict	Dict. of state arrays	required
X	ndarray	INSERT DESCRIPTION	required
local_mask_info	dict	INSERT DESCRIPTION	required
obs_data	ndarray	Observed data	required
pred_data	ndarray	Predicted data	required
parallel	int	Number of parallel runs	required
actnum	ndarray	Active cells	None
field_dim	list	Number of grid cells in each direction	None
act_data_list	list	List of active data names	None
scale_data	ndarray	Scaling array for data	None
num_states	int	Number of states	1
emp_d_cov	bool	INSERT DESCRIPTION	False

Notes

Since the localization matrix is to large for evaluation, we instead calculate it row for row.

resample_state(aug_state, state, list_state, new_en_size)

Extract the seperate state variables from an augmented state matrix. Calculate the mean and covariance, and resample this.

Parameters:

Name	Type	Description	Default
aug_state	ndarray	Augmented matrix of state variables	required
state	dict	Dict. af state variables	required
list_state	list	List of state variable	required
new_en_size	int	Size of the new ensemble	required

Returns:

Name	Type	Description
state	dict	Dict. of resampled members

save_analysisdebug(ind_save, **kwargs)

Save variables in analysis step for debugging purpose

Parameters:

Name	Type	Description	Default
ind_save	int	Index of analysis step	required
**kwargs	dict	Variables that will be saved to npz file	{}

Notes

Use kwargs here because the input will be a dictionary with names equal the variable names to store, and when this is passed to np.savez (kwargs) the variable will be stored with their original name.

screen_data(cov_data, pred_data, obs_data_vector, keys_da, iteration)

INSERT DESCRIPTION

Parameters:

Name	Type	Description	Default
cov_data	ndarray	Data covariance matrix	required
pred_data	ndarray	Predicted data	required
obs_data_vector		Observed data (1D array)	required
keys_da	dict	Dictionary with every input in DATAASSIM	required
iteration	int	Current iteration	required

Returns:

Name	Type	Description
cov_data	ndarray	Updated data covariance matrix

store_ensemble_sim_information(saveinfo, member)

Here, we can either run a unique python script or do some other post-processing routines. The function should not return anything, but provide a method for storing revevant information. Input the current member for easy storage

subsample_state(index, aug_state, pert_state)

Draw a subsample from the original state, given by the index

Parameters:

Name	Type	Description	Default
index	ndarray	Index of parameters to draw.	required
aug_state	ndarray	Original augmented state.	required
pert_state	ndarray	Perturbed augmented state, for error covariance.	required

Returns:

Name	Type	Description
new_state	dict	Subsample of state.

update_datavar(cov_data, datavar, assim_index, list_data)

Extract the separate variance from an augmented vector. It is assumed that the augmented variance is made gen_covdata, hence this is the reverse method of gen_covdata.

Parameters:

Name	Type	Description	Default
cov_data	array - like	Augmented vector of variance.	required
datavar	dict	Dictionary of separate variances.	required
assim_index	list	Assimilation order as a list.	required
list_data	list	List of data keys.	required

Returns:

Name	Type	Description
datavar	dict	Updated dictionary of separate variances.

update_state(aug_state, state, list_state, cell_index=None)

Extract the separate state variables from an augmented state array. It is assumed that the augmented state array is made in aug_state, hence this is the reverse method of aug_state.

Parameters:

Name	Type	Description	Default
aug_state	ndarray	Augmented array of UPDATED state variables	required
state	dict	Dict. of state variables NOT updated.	required
list_state	list	List of state keys that have been updated	required
cell_index	list	List of indexes that gives the where the aug state should be placed	None

Returns:

Name	Type	Description
state	dict	Dict. of UPDATED state variables