Reconciliation module
OpenSEFA.ReconciliationModule — Module
Core types and algorithms for data reconciliation.
Data reconciliation finds the most likely values of measured quantities subject to model constraints by minimizing a weighted sum of squared adjustments (a least-squares problem).
Key types:
ReconciliationProblem— holds variables (measured, unmeasured, fixed) and constraint equations.ReconciliationSolution— result of a solve call, including variable values and solution status.ReconciliationSolver— abstract type for solver backends.ReconciliationPipeline— abstract type for multi-stage solution pipelines.
OpenSEFA.ReconciliationModule.SUCCESS_CODES — Constant
Any of the following is considered a successful solve.
OpenSEFA.ReconciliationModule.AbstractReconciliationProblem — Type
Abstract type for reconciliation problems.
Allows different problem types (e.g., ReconciliationProblem, OptimizationProblem) to share common solver infrastructure while maintaining type safety.
OpenSEFA.ReconciliationModule.EID — Type
Index for a given equation. Indices are unique within the same system.
OpenSEFA.ReconciliationModule.ReconciliationPipeline — Type
Extend this type for solver pipelines.
OpenSEFA.ReconciliationModule.ReconciliationProblem — Type
Contains various dictionaries that hold problem data for efficient lookups.
To create an empty problem, use ReconciliationProblem(). The following interface functions should be used to act upon reconciliation problems (instead of operations on the arrays directly):
add_variable!– Add a variable to the problem.remove_variable!– Remove a variable from the problem.add_equation!– Add an equation to the problem.remove_equation!– Remove an equation from the problem.
Each of these methods works for an individual variable or an array.
OpenSEFA.ReconciliationModule.ReconciliationSolution — Type
Holds the reconciliation problem as well as values after variable reconciliation. Such values can be used to judge the quality of the solution.
OpenSEFA.ReconciliationModule.ReconciliationSolver — Type
Extend this type for solver backends.
OpenSEFA.ReconciliationModule.STATUS — Type
Status returned by the solver.
FEASIBLE: All variables have assigned value (no unobservable variables)PARTIALLY_FEASIBLE: Some variables have assigned value (from calling Presolve)FEASIBLE_WITH_UNOBSERVABLE_VARIABLES: There are some variables which are unobservableCONTRADICTORY: Cases for which we find infeasibility after variable replacementsINFEASIBLE: Cases in which the solver was not able to find the solutionUNKNOWN: Default value for methods (never returned by a method)
OpenSEFA.ReconciliationModule.SolutionComparison — Type
Holds the result of the comparision of two ReconciliationSolutions.
OpenSEFA.ReconciliationModule.SolutionScore — Type
Holds scores that describe the quality of a ReconciliationSolution.
OpenSEFA.ReconciliationModule.SparseMatrix — Type
Convenience alias for sparse matrices in compressed sparse column format.
Base.convert — Method
Convert a STANTrace into a reconciliation problem.
CommonSolve.solve! — Method
Extend this method for specific solvers.
OpenSEFA.ConstraintsModule.evaluate — Method
Evaluate the equations of the reconciliation problem. Returns the numerical evaluation of the left-hand side of each equation (it is expected to be zero).
OpenSEFA.ConstraintsModule.is_bilinear — Method
Return if the system has at least one bilinear equation.
OpenSEFA.ConstraintsModule.is_linear — Method
Return if the system only has linear flow equations.
OpenSEFA.ConstraintsModule.is_transfer_coefficient — Method
Return if the named variable represents a transfer coefficient.
OpenSEFA.ErrorInfosModule.add_info! — Method
Add an info to a ReconciliationSolution. Check if the info key already exists.
OpenSEFA.ReconciliationModule.add_bilinear_term! — Function
Add a biinear term 'factor * var1 * var2' to the equation 'eid' of a ReconciliationProblem 'rec'. If a bilinear term with variables 'var1' and 'var2' existed before but cancels with the added bilinear term, the variables 'var1' and 'var2' are stored in 'removed_variables'.
OpenSEFA.ReconciliationModule.add_equation! — Method
Add an equation to the reconciliation problem.
If strict is true (default) an error is thrown unless all of the following conditions hold:
- All variables in the equation need to exist in the problem (see
add_variable!), otherwise an error is thrown. - All linear terms have distinct names.
- All bilinear terms have distinct names, modulo squared terms and permutations within the same term.
Such assumptions are useful to improve the efficiency of certain algorithms.
OpenSEFA.ReconciliationModule.add_linear_term! — Function
Add a linear term 'factor * var' to the equation 'eid' of a ReconciliationProblem 'rec'. If a linear term with variable 'var' existed before but cancels with the added linear term, the variable 'var' is stored in 'removed_variables'.
OpenSEFA.ReconciliationModule.add_new_bilinear_term! — Method
Add a bilinear term 'factor * var1 * var2' to the equation 'eid' of a ReconciliationProblem 'rec' assuming that there is no bilinear term with variables 'var1' and 'var2' in equation 'eid' yet.
OpenSEFA.ReconciliationModule.add_new_linear_term! — Method
Add a linear term 'factor * var' to the equation 'eid' of a ReconciliationProblem 'rec' assuming that there is no linear term with variable 'var' in equation 'eid' yet.
OpenSEFA.ReconciliationModule.add_to_bilinear_term! — Function
Add 'value' to the factor of the BilinearTerm with index 'btindex' of the equation 'eid' of a ReconciliationProblem 'rec'. If the added 'value' and the previous factor cancel each other, the BilinearTerm is removed from the equation and the variables stored in 'removedvariables'. Note that the dictionaries rec.vartoeq and rec.eqtovar and rec.variables are not updated. (Requires further checking using 'removed_variables'.)
OpenSEFA.ReconciliationModule.add_to_linear_term! — Function
Add 'value' to the factor of the LinearTerm with index 'ltindex' of the equation 'eid' of a ReconciliationProblem 'rec'. If the added 'value' and the previous factor cancel each other, the LinearTerm is removed from the equation and the variable stored in 'removedvariables'. Note that the dictionaries rec.vartoeq and rec.eqtovar and rec.variables are not updated. (Requires further checking using 'removed_variables'.)
OpenSEFA.ReconciliationModule.add_variable! — Method
Add a new variable to the reconciliation problem.
OpenSEFA.ReconciliationModule.check_residuals — Method
Checks for a given SolutionScore if every equation satisfies either absoluteresidual < atol or relativeresidual < rtol.
OpenSEFA.ReconciliationModule.compare — Method
Compares two provided ReconciliationSolutions. It is assumed that the underlying ReconciliationProblems are identical. (This is roughly checked by comparing the problem names, the variable names and the types of shared variables.) For each variable that has assigned values in both ReconciliationSolutions, the absolute difference of the assigned values and the relative difference (absolute difference divided by the mean of the assigned values) is returned. For each equation that has assigned values in both ReconciliationSolutions, the absolute difference of the assigned values is returned. (Note that two equations are considered to be the same only if they have exactly the same form, including the order of variables etc.)
OpenSEFA.ReconciliationModule.compute_relative_residual — Method
Compute the relative residual for a single equation.
The relative residual is |r(x)| / (|c| + ||q|| ||x|| + ||P|| ||x||^2) where:
r(x)is the residual (signed equation value)cis the constant termqare the linear term coefficientsPare the bilinear term coefficients
This computation relies on equations_value, i.e., it can only be computed if the equation can be evaluated. This typically corresponds to all variables having assigned values. As an exception to such rule, for a bilinear term 'factor * var1 * var2' we allow that a variable is unassigned if the other is zero.
Note: The constraint equation does not define P uniquely but only P{ij} + P{ji} for indices i, j. We always set P such that for i != j it holds either P{ij} == 0 or P{ji} == 0, but theoretically it could be done in a different way (e.g., defining P to be symmetric).
The formula uses ||x|| computed from ALL unique variables appearing in the equation. This ensures consistency: the same variable vector is used for both linear and bilinear contributions.
This function computes the denominator directly from the equation structure without QCQP conversion.
OpenSEFA.ReconciliationModule.evaluate_fixed_discrepancy — Method
Evaluate the fixed constraints (fixed variables of the reconciliation problem). Returns the element-wise absolute difference between expected (fixed) values and results.
OpenSEFA.ReconciliationModule.evaluate_globality — Method
Evaluates if it can be proven that any local solution of a given ReconciliationProblem is global.
OpenSEFA.ReconciliationModule.evaluate_objective — Method
Evalutes the weighted least-squares function (x - xm)^T * Qinv * (x - xm) between measured and reconciled variables, where Q is the matrix of variance-covariance uncertainties. If the problem has no measured variables, nothing is returned.
OpenSEFA.ReconciliationModule.evaluate_score — Function
Given the initial problem and a dictionary of assigned values (eg. result of optimization), then this function estimates the percentage of variables (and equations) whose value is assigned.
OpenSEFA.ReconciliationModule.get_reconciliation_problem — Method
get_reconciliation_problem(problem::AbstractReconciliationProblem)Extract the underlying ReconciliationProblem from any AbstractReconciliationProblem. For ReconciliationProblem itself, returns self. For wrapper types like OptimizationProblem, this must be overloaded to extract the wrapped ReconciliationProblem.
OpenSEFA.ReconciliationModule.get_variable — Method
Retrieve the variable instance from a name, or nothing if the variable is not in the problem.
OpenSEFA.ReconciliationModule.get_variable_indices — Method
Assign each measured, unmeasured and fixed variables (in that order) to an integer index starting at 1.
OpenSEFA.ReconciliationModule.has_basic_constraints — Method
Check if the solution satisfies the basic constraints:
- All measured and unmeasured variables have assigned nonnegative values.
- All transfer coefficients have assigned a value in the
[0, 1]interval. - Upper and lower bounds for each variable are respected.
OpenSEFA.ReconciliationModule.initialize_variable_map — Method
Each variable is mapped to the indices of the equations in which it participates (ie. it is in a linear, bilinear term or both).
OpenSEFA.ReconciliationModule.is_cost_functional_supported — Method
By default assume that cost functionals are not supported.
OpenSEFA.ReconciliationModule.jacobian — Function
Computes Jacobian matrix for a ReconciliationProblem at a given point p as a dense matrix. It returns matrices (; Jx, Jy, Jz, b), where each row is an equation and each column is a variable (measured in Jx, unmeasured in Jy, fixed in Jz) or the constant terms (b), i.e., the equation system is approximated by the linearization [Jx Jy Jz] * [x^T, y^T, z^T]^T = -b for measured variables x, unmeasured variables y and fixed variables z. Computing the Jacobian matrix for bilinear terms requires the evaluation at some p. If p doesn't provide the evaluation values for a bilinear term, the equation including it is skipped and not part of Jx, Jy, Jz and b.
OpenSEFA.ReconciliationModule.jacobians — Method
Compute the Jacobians Jx and Jy for measured and unmeasured variables respectively.
The dictionary variable_index maps variable names to indices in the full matrix [Jx Jy]. For efficiency, this method assumes that the problem has no fixed variables. If that is not the case you should apply remove_fixed_variables! as pre-processing step.
OpenSEFA.ReconciliationModule.jacobians_and_hessian — Method
Compute Jacobians and Hessian for a reconciliation problem.
The theoretical derivation can be found in resources/error_propagation.qmd.
OpenSEFA.ReconciliationModule.remove_equation! — Method
Remove an equation from the reconciliation problem. The method does not perform any variable removals.
OpenSEFA.ReconciliationModule.remove_variable! — Method
Remove a variable from the reconciliation problem.
Note: removing a variable does not affect the equations in which it may participate; those need to be adjusted accordingly.
In some cases we use this method to workaround the fact that variable structs are immutable. For example, in order to change the bound of a variable we can add it and remove one with the same name. By doing so, since the name is the same the coupling with the equations in which it participates should remain unchanged.
OpenSEFA.ReconciliationModule.solver_max_iter — Method
Return the maximum iteration count for logging, or nothing if not applicable.
OpenSEFA.ReconciliationModule.solver_succeeded — Method
Whether the solver succeeded to produce an answer (total or partial).
OpenSEFA.ReconciliationModule.solver_tolerance — Method
Return the solver tolerance for logging, or nothing if not applicable.
OpenSEFA.ReconciliationModule.update_vars_dicts! — Function
Check for variables in 'varstocheck' and equations in 'eqstocheck' if they have to be removed from vartoeq and eqtovar in the ReconciliationProblem 'rec'. If necessary, remove from the dictionaries and store removed variables in 'removedvariables'. Note that it is not checked if a variable is unconstrained/free. (Requires further checking using 'removedvariables'.) If 'varstocheck' or 'eqstocheck' is not specified (i.e., 'nothing'), check all variables and equations from the ReconciliationProblem.
OpenSEFA.ReconciliationModule.weights_matrix — Method
Return the weights matrix Q containing variance-covariance components of the measured quantities. If the problem has no measured variables, nothing is returned.