Reconciliation module

OpenSEFA.ReconciliationModule — Module

Core types and algorithms for data reconciliation.

Data reconciliation finds the most likely values of measured quantities subject to model constraints by minimizing a weighted sum of squared adjustments (a least-squares problem).

Key types:

  • ReconciliationProblem — holds variables (measured, unmeasured, fixed) and constraint equations.
  • ReconciliationSolution — result of a solve call, including variable values and solution status.
  • ReconciliationSolver — abstract type for solver backends.
  • ReconciliationPipeline — abstract type for multi-stage solution pipelines.
source
OpenSEFA.ReconciliationModule.ReconciliationProblem — Type

Contains various dictionaries that hold problem data for efficient lookups.

To create an empty problem, use ReconciliationProblem(). The following interface functions should be used to act upon reconciliation problems (instead of operations on the arrays directly):

  • add_variable! – Add a variable to the problem.
  • remove_variable! – Remove a variable from the problem.
  • add_equation! – Add an equation to the problem.
  • remove_equation! – Remove an equation from the problem.

Each of these methods works for an individual variable or an array.

source
OpenSEFA.ReconciliationModule.STATUS — Type

Status returned by the solver.

  • FEASIBLE : All variables have assigned value (no unobservable variables)
  • PARTIALLY_FEASIBLE : Some variables have assigned value (from calling Presolve)
  • FEASIBLE_WITH_UNOBSERVABLE_VARIABLES : There are some variables which are unobservable
  • CONTRADICTORY : Cases for which we find infeasibility after variable replacements
  • INFEASIBLE : Cases in which the solver was not able to find the solution
  • UNKNOWN : Default value for methods (never returned by a method)
source
OpenSEFA.ConstraintsModule.evaluate — Method

Evaluate the equations of the reconciliation problem. Returns the numerical evaluation of the left-hand side of each equation (it is expected to be zero).

source
OpenSEFA.ReconciliationModule.add_bilinear_term! — Function

Add a biinear term 'factor * var1 * var2' to the equation 'eid' of a ReconciliationProblem 'rec'. If a bilinear term with variables 'var1' and 'var2' existed before but cancels with the added bilinear term, the variables 'var1' and 'var2' are stored in 'removed_variables'.

source
OpenSEFA.ReconciliationModule.add_equation! — Method

Add an equation to the reconciliation problem.

If strict is true (default) an error is thrown unless all of the following conditions hold:

  • All variables in the equation need to exist in the problem (see add_variable!), otherwise an error is thrown.
  • All linear terms have distinct names.
  • All bilinear terms have distinct names, modulo squared terms and permutations within the same term.

Such assumptions are useful to improve the efficiency of certain algorithms.

source
OpenSEFA.ReconciliationModule.add_linear_term! — Function

Add a linear term 'factor * var' to the equation 'eid' of a ReconciliationProblem 'rec'. If a linear term with variable 'var' existed before but cancels with the added linear term, the variable 'var' is stored in 'removed_variables'.

source
OpenSEFA.ReconciliationModule.add_to_bilinear_term! — Function

Add 'value' to the factor of the BilinearTerm with index 'btindex' of the equation 'eid' of a ReconciliationProblem 'rec'. If the added 'value' and the previous factor cancel each other, the BilinearTerm is removed from the equation and the variables stored in 'removedvariables'. Note that the dictionaries rec.vartoeq and rec.eqtovar and rec.variables are not updated. (Requires further checking using 'removed_variables'.)

source
OpenSEFA.ReconciliationModule.add_to_linear_term! — Function

Add 'value' to the factor of the LinearTerm with index 'ltindex' of the equation 'eid' of a ReconciliationProblem 'rec'. If the added 'value' and the previous factor cancel each other, the LinearTerm is removed from the equation and the variable stored in 'removedvariables'. Note that the dictionaries rec.vartoeq and rec.eqtovar and rec.variables are not updated. (Requires further checking using 'removed_variables'.)

source
OpenSEFA.ReconciliationModule.compare — Method

Compares two provided ReconciliationSolutions. It is assumed that the underlying ReconciliationProblems are identical. (This is roughly checked by comparing the problem names, the variable names and the types of shared variables.) For each variable that has assigned values in both ReconciliationSolutions, the absolute difference of the assigned values and the relative difference (absolute difference divided by the mean of the assigned values) is returned. For each equation that has assigned values in both ReconciliationSolutions, the absolute difference of the assigned values is returned. (Note that two equations are considered to be the same only if they have exactly the same form, including the order of variables etc.)

source
OpenSEFA.ReconciliationModule.compute_relative_residual — Method

Compute the relative residual for a single equation.

The relative residual is |r(x)| / (|c| + ||q|| ||x|| + ||P|| ||x||^2) where:

  • r(x) is the residual (signed equation value)
  • c is the constant term
  • q are the linear term coefficients
  • P are the bilinear term coefficients

This computation relies on equations_value, i.e., it can only be computed if the equation can be evaluated. This typically corresponds to all variables having assigned values. As an exception to such rule, for a bilinear term 'factor * var1 * var2' we allow that a variable is unassigned if the other is zero.

Note: The constraint equation does not define P uniquely but only P{ij} + P{ji} for indices i, j. We always set P such that for i != j it holds either P{ij} == 0 or P{ji} == 0, but theoretically it could be done in a different way (e.g., defining P to be symmetric).

The formula uses ||x|| computed from ALL unique variables appearing in the equation. This ensures consistency: the same variable vector is used for both linear and bilinear contributions.

This function computes the denominator directly from the equation structure without QCQP conversion.

source
OpenSEFA.ReconciliationModule.evaluate_objective — Method

Evalutes the weighted least-squares function (x - xm)^T * Qinv * (x - xm) between measured and reconciled variables, where Q is the matrix of variance-covariance uncertainties. If the problem has no measured variables, nothing is returned.

source
OpenSEFA.ReconciliationModule.evaluate_score — Function

Given the initial problem and a dictionary of assigned values (eg. result of optimization), then this function estimates the percentage of variables (and equations) whose value is assigned.

source
OpenSEFA.ReconciliationModule.get_reconciliation_problem — Method
get_reconciliation_problem(problem::AbstractReconciliationProblem)

Extract the underlying ReconciliationProblem from any AbstractReconciliationProblem. For ReconciliationProblem itself, returns self. For wrapper types like OptimizationProblem, this must be overloaded to extract the wrapped ReconciliationProblem.

source
OpenSEFA.ReconciliationModule.has_basic_constraints — Method

Check if the solution satisfies the basic constraints:

  • All measured and unmeasured variables have assigned nonnegative values.
  • All transfer coefficients have assigned a value in the [0, 1] interval.
  • Upper and lower bounds for each variable are respected.
source
OpenSEFA.ReconciliationModule.jacobian — Function

Computes Jacobian matrix for a ReconciliationProblem at a given point p as a dense matrix. It returns matrices (; Jx, Jy, Jz, b), where each row is an equation and each column is a variable (measured in Jx, unmeasured in Jy, fixed in Jz) or the constant terms (b), i.e., the equation system is approximated by the linearization [Jx Jy Jz] * [x^T, y^T, z^T]^T = -b for measured variables x, unmeasured variables y and fixed variables z. Computing the Jacobian matrix for bilinear terms requires the evaluation at some p. If p doesn't provide the evaluation values for a bilinear term, the equation including it is skipped and not part of Jx, Jy, Jz and b.

source
OpenSEFA.ReconciliationModule.jacobians — Method

Compute the Jacobians Jx and Jy for measured and unmeasured variables respectively.

The dictionary variable_index maps variable names to indices in the full matrix [Jx Jy]. For efficiency, this method assumes that the problem has no fixed variables. If that is not the case you should apply remove_fixed_variables! as pre-processing step.

source
OpenSEFA.ReconciliationModule.remove_variable! — Method

Remove a variable from the reconciliation problem.

Note: removing a variable does not affect the equations in which it may participate; those need to be adjusted accordingly.

In some cases we use this method to workaround the fact that variable structs are immutable. For example, in order to change the bound of a variable we can add it and remove one with the same name. By doing so, since the name is the same the coupling with the equations in which it participates should remain unchanged.

source
OpenSEFA.ReconciliationModule.update_vars_dicts! — Function

Check for variables in 'varstocheck' and equations in 'eqstocheck' if they have to be removed from vartoeq and eqtovar in the ReconciliationProblem 'rec'. If necessary, remove from the dictionaries and store removed variables in 'removedvariables'. Note that it is not checked if a variable is unconstrained/free. (Requires further checking using 'removedvariables'.) If 'varstocheck' or 'eqstocheck' is not specified (i.e., 'nothing'), check all variables and equations from the ReconciliationProblem.

source