Unit system overhaul#217
Conversation
Incorporated docstring examples from develop while preserving the unit-system-overhaul API (x_unit/y_unit split). Kept HEAD repr format and API throughout; updated ExpressionComponent class docstring example from unit= to x_unit=. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Develop's cleanly-merged changes introduced self.unit (old API) in __repr__ methods of ResolutionModel, Convolution, NumericalConvolution, AnalyticalConvolution, and BackgroundModel. Updated all to use self.x_unit. Added missing __init__ docstrings to ResolutionModel and SampleModel. Updated test assertions to match HEAD repr format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers previously untested branches added in the unit-system overhaul: - FitBinding: string modes in __init__, delta mode in _build_diffusion_callable - ConvolutionBase: convert_x/y_unit when components are None - ParameterAnalysis: no-variance and all-NaN-variance paths in _get_xyweight_from_dataset; None x_unit and None y_unit in _get_unit_conversions - Experiment: load_hdf5 TypeError for non-DataArray return value; get_finite_energy_mask and get_masked_binned_data with and without data - Analysis / Analysis1d: __repr__ methods Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…backing-field reads with property calls When _x_unit or _y_unit was None, ModelComponent.x_unit returned the string 'None', making unit-None checks always evaluate as truthy. Fixed the properties to return None when the backing field is None, then replaced all ~90 direct self._x_unit/self._y_unit reads across 13 source files with the property so the fix propagates everywhere. Also corrects the InstrumentModel.x_unit annotation (adds | None) and upgrades two SimpleNamespace test mocks to real Polynomial objects now that the property behaves correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
henrikjacobsenfys
left a comment
There was a problem hiding this comment.
Review of all files, but not tests
|
|
||
| **Physical constants** | ||
|
|
||
| The symbols ``hbar`` (in meV*ps) and ``kb`` (in meV/K) are provided automatically as read-only |
There was a problem hiding this comment.
I think meV*s makes more sense
| parameter_units : dict[str, str | sc.Unit] | None, default=None | ||
| Optional units per parameter name. Each entry sets the unit of the named parameter | ||
| without rescaling its value (see :meth:`set_unit`), and takes precedence over the unit | ||
| of a Parameter instance given in *parameters*. When units are in use, a warning is |
There was a problem hiding this comment.
Warning or error?
| return candidate | ||
| return unit | ||
|
|
||
| def _propagate_unit(self, node: sp.Basic) -> sc.Unit: |
There was a problem hiding this comment.
Need to check carefully
| center_param = Parameter(name=name + ' center', value=float(center), unit=unit) | ||
| if enforce_minimum_center and center_param.min < DHO_MINIMUM_CENTER: | ||
| center_param.min = DHO_MINIMUM_CENTER | ||
| if center_param.value < DHO_MINIMUM_CENTER: |
There was a problem hiding this comment.
Can this happen? Think about this for a bit.
Perhaps set min to -inf, then set value, then update min?
| Unit for the y-axis (dependent variable / output) of this component. | ||
| name : str, default='ModelComponent' | ||
| The name of the model component for indexing. | ||
| Internal name used for parameter labelling and logging. |
There was a problem hiding this comment.
Ahhh not more of these
| @unit.setter | ||
| def unit(self, _unit_str: str) -> None: | ||
| @x_unit.setter | ||
| def x_unit(self, _: str) -> None: |
There was a problem hiding this comment.
Could x_unit and y_unit properties be moved to base classes?
| Unit of the model. | ||
| x_unit : str | sc.Unit, default='meV' | ||
| Unit of the x-axis. | ||
| y_unit : str | sc.Unit, default='dimensionless' |
There was a problem hiding this comment.
Does y_unit make sense? Or should it always be dimensionless, even if it's created from a samplemodel with dimensions?
| f'Rename them to have unique names before removing.' | ||
| ) | ||
| del self.diffusion_models[matches[0]] | ||
| self._component_collections_is_dirty = True |
There was a problem hiding this comment.
Hmm
Deleting a model should not mean we rebuild the component collection- we should just remove the corresponding components everywhere. This should be made into an issue for later
| # version at the last explicit convolution_plan_is_valid = True, which suppresses | ||
| # rebuilds for all convolvers until the next invalidation. | ||
| self._plan_version = 0 | ||
| self._blessed_plan_version = -1 |
There was a problem hiding this comment.
Is this blessing used anywhere?
| Q_type = np.ndarray | Numeric | list | ArrayLike | sc.Variable | ||
| energy_type = np.ndarray | Numeric | list | ArrayLike | sc.Variable | ||
|
|
||
| hbar = DescriptorNumber.from_scipp('hbar', scipp_hbar) |
There was a problem hiding this comment.
Use these in expression component
henrikjacobsenfys
left a comment
There was a problem hiding this comment.
Review some tests
There was a problem hiding this comment.
Should not be rebuilt, outcomment relevant parts of notebook
| GAUSSIAN_UNITS = {'A': 'meV', 'x0': 'meV', 'sigma': 'meV'} | ||
|
|
||
|
|
||
| class TestExpressionComponentUnitCorrectness: |
There was a problem hiding this comment.
New class? Not sure
|
|
||
| def test_output_unit_matching_y_unit_does_not_warn(self): | ||
| # WHEN THEN EXPECT: declaring the matching y_unit silences the mismatch warning | ||
| with warnings.catch_warnings(): |
There was a problem hiding this comment.
What does this do?
|
|
||
| def test_output_unit_preserved_by_abs(self): | ||
| # WHEN | ||
| with pytest.warns(UserWarning, match='does not match'): |
There was a problem hiding this comment.
Does it warn? Need to check
| ) | ||
| x = sc.linspace('energy', -1.0, 1.0, 11, unit='ueV') | ||
|
|
||
| # THEN EXPECT: ExpressionComponent cannot auto-convert, so it warns and uses raw values |
There was a problem hiding this comment.
Is that desired or should it error?
| x = sc.linspace('energy', -1.0, 1.0, 11, unit='\u00b5eV') | ||
|
|
||
| # THEN EXPECT: no spurious warning from the differing spellings | ||
| with warnings.catch_warnings(): |
There was a problem hiding this comment.
What does this do?
|
|
||
| # THEN EXPECT: relabelling A breaks the output unit | ||
| with pytest.warns(UserWarning, match='does not match'): | ||
| expr.set_unit('A', '1/ueV') |
There was a problem hiding this comment.
This should match, no?
| ) | ||
|
|
||
| # THEN EXPECT: constants are not fittable variables | ||
| names = {p.name for p in expr.get_all_variables()} |
There was a problem hiding this comment.
Shouldn't they be in all variables?
| expr.kb.convert_unit('eV/K') | ||
|
|
||
| # EXPECT: the value is rescaled with the unit | ||
| assert expr.kb.value == pytest.approx(8.617333e-05, rel=1e-6) |
There was a problem hiding this comment.
It shouldn't ever need necessary to convert the unit of constants - only output unit matters.
Thought: use convert to make sure output unit matches y_unit? Otherwise it may become J and y_unit be meV or something silly like that
There was a problem hiding this comment.
Add tests of suppress warnings?
PR: Unit system overhaul
Branch:
unit-system-overhaul→developSummary
The single
unitattribute on all model objects is replaced by two separate attributes,x_unit(independent variable / energy axis) andy_unit(model output / intensity). This distinction was needed to correctly handle parameter units, area normalisation, and unit conversion inParameterAnalysis. Several pre-existing bugs were found and fixed during the work.Unit system changes
EasyDynamicsModelBaseThe single
unitparameter and property is replaced byx_unit(default'meV') andy_unit(default'dimensionless'). Both properties are read-only via setters that raiseAttributeError; the corresponding mutation methods areconvert_x_unitandconvert_y_unit. The backing fields_unit→_x_unit/_y_unitare updated everywhere in the hierarchy.ModelComponentand all concrete componentsx_unitandy_unitare threaded throughLorentzian,Gaussian,DeltaFunction,Exponential,Voigt,DampedHarmonicOscillator,Polynomial, andExpressionComponent. Each component gainsconvert_x_unitandconvert_y_unitmethods with rollback on failure.The
evaluate(x)method on all components gains anoutputkeyword ('numpy'by default,'scipp'to return asc.Variablewithy_unitattached)._prepare_x_for_evaluatenow returns a(x_vals, detected_unit, dim)tuple instead of just the raw values. A new helper_resolve_param_valueconverts each parameter into the unit detected fromxat call time, so components handle mixed units transparently without mutating any stored parameters.Area parameter unit is now
x_unit * y_unitPreviously
areawas stored in the same unit as the energy axis. TheCreateParametersMixin._create_area_parameterhelper now derives the area unit asstr(sc.Unit(x_unit) * sc.Unit(y_unit)). For the defaulty_unit='dimensionless'this is unchanged, but it makes the area semantics correct when a physicaly_unitis set.ComponentCollectionconvert_unitis split intoconvert_x_unitandconvert_y_unit, both with atomic rollback. Theevaluateandevaluate_componentmethods gain the sameoutputkeyword.to_dict/from_dictare updated accordingly ('unit'key →'x_unit'/'y_unit').ModelBase,SampleModel,InstrumentModel,ResolutionModel,BackgroundModelAll accept
x_unit/y_unitconstructor arguments and propagateconvert_x_unit/convert_y_unitdown to theirComponentCollectionchildren.Convolution stack (
ConvolutionBase,NumericalConvolutionBase,Convolution,AnalyticalConvolution,NumericalConvolution)unit→x_unit/y_unitthroughout.ConvolutionBase.convert_x_unitis a transaction that converts the energy array, energy offset, and both component collections atomically (with rollback). A newConvolutionBase.convert_y_unitpropagates the new unit to the sample components only (resolution is dimensionless by construction).Convolution.convert_y_unitadditionally propagates to its internal analytical and numerical sub-convolvers.DiffusionModelBaseand diffusion modelsunit→x_unit/y_unitsignature throughoutDiffusionModelBase,DeltaLorentz,BrownianTranslationalDiffusion, andJumpTranslationalDiffusion.Analysis1dReferences to
self.unit(the old single unit) replaced byself.x_unitwherever the energy axis is meant.ParameterAnalysisA new private method
_get_unit_conversions(binding, pname)computes(x_factor, y_factor)scale factors that convert the stored Q-coordinate and parameter units into the model's declaredx_unit/y_unit. The factors are applied before passing data to the model callable and to the fit, and are also applied when building the model overlay curves.DiffusionModelBasebindings always return(1.0, 1.0)because their callables take raw Q values.utils.pyTwo new public helpers:
verify_Q_index(Q_index, Q)(shared validation extracted fromExperimentandAnalysisBase);energy_to_scipp(energy, unit)(converts a raw numpy array into asc.Variableconsistently).Experiment_extract_x_y_weights_only_finiteis made public (extract_x_y_weights_only_finite). Two new public methods are added:get_finite_energy_mask(Q_index)returns a booleansc.Variableselecting finite points;get_masked_binned_data(Q_index)returns the single-Q data slice with non-finite points removed.Bug fixes
1.
ModelComponent.x_unit/y_unitreturning'None'as a stringWhat broke:
x_unitandy_unitproperties onModelComponentwere reading the old backing field_unitdirectly viastr(self._unit). After the rename to_x_unit/_y_unitthe field wasNone, so the property returned the literal string'None'instead ofNone.Fix: The properties now guard with
str(self._x_unit) if self._x_unit is not None else None.Regression test:
test_y_unit_default,test_y_unit_setter_raisesadded forModelComponentand every concrete component class.2.
ParameterAnalysis.fitandcalculate_model_datasetignoring unit mismatchesWhat broke: When fitting a Q-dependent model (e.g. a
Lorentziandeclared withx_unit='1/angstrom') against extracted parameters stored in different units (e.g. Q in'1/m'), the raw stored values were passed directly to the model callable without any conversion. This silently produced numerically wrong fits by several orders of magnitude.Fix:
_get_unit_conversionscomputes the correctx_factorandy_factorusingsc.to_unit; both methods apply the factors before fitting and before building overlays.Regression tests:
test_fit_converts_x_unit,test_fit_converts_y_unit,test_calculate_model_dataset_converts_x_unit,test_calculate_model_dataset_converts_y_unit,test_fit_incompatible_x_unit_raises,test_fit_incompatible_y_unit_raisesintest_parameter_analysis.py.3.
ParameterAnalysis._get_xyweight_from_datasetcrashing on NaN variancesWhat broke: When only a subset of Q values has a fitted value for a given parameter,
parameters_to_datasetfillsnp.nanfor the missing rows. The old weight calculation sawnp.nanvariance and raisedValueError('Non-finite variances found'), making it impossible to fit Q-dependent models over partial Q ranges.Fix: NaN variances are now identified separately (
nan_mask) and filtered silently. Only non-NaN non-finite or non-positive variances raise; only the valid rows are returned as(q_values[valid_mask], values[valid_mask], 1/sqrt(variances[valid_mask])).Regression tests:
test_get_xyweight_from_dataset_nan_variance_filters_row,test_get_xyweight_from_dataset_all_nan_variances_raisesintest_parameter_analysis.py.4.
FitBinding._get_parameter_namesreturning wrong names when 'delta' was in the mode listWhat broke: For
DiffusionModelBasebindings, if'delta'appeared anywhere in the mode list, the method returned'{name} area'for every mode in the list — including non-delta modes like'lorentzian'. This caused the fitter to look for a parameter named e.g.'D area'instead of'D lorentzian', making diffusion fitting silently broken.Fix: The special
if 'delta' in modesbranch is removed. The method now simply returns[f'{name} {mode}' for mode in modes]for all diffusion models.Regression test:
test_build_diffusion_callable_delta_modeintest_fit_binding.py.5.
FitBindingtreating a bare stringmodesas a character iteratorWhat broke:
FitBinding(parameter_name='D', model=model, modes='lorentzian')— passing a single string instead of a list — caused the mode list to be iterated character by character (['l', 'o', 'r', ...]) rather than treated as a single mode.Fix: A guard added in
__init__:if isinstance(modes, str): modes = [modes].Regression test:
test_init_with_string_modesintest_fit_binding.py.6. Energy offset unit mismatch raising an error for compatible units
What broke:
Analysis1d._apply_energy_offsetcheckedenergy.unit != energy_offset.unitand, if different, attemptedenergy_offset.convert_unit(str(energy.unit))— which permanently mutated the storedenergy_offsetparameter. If units were compatible but not identical (e.g. energy grid inmeV, offset parameter ineV), this silently changed the parameter's unit, and in the convolution layer the sameenergy_offset.valuewas subtracted without conversion.Fix: Both
Analysis1d._apply_energy_offsetandConvolutionBase._energy_with_offsetnow usesc.to_unit(energy_offset.full_value, energy.unit).value— a non-mutating conversion — so the stored parameter is never changed.Regression tests:
test_energy_with_offset_unit_conversionintest_convolution_base.py;test_convolution_with_energy_offset_in_different_unitintest_numerical_convolution.py.7.
Analysis1d.calculatemutating the stored convolverWhat broke:
calculate(energy=...)(the public plotting path) overwroteself._convolverwith a convolver built for the plotting energy grid, then immediately setself._convolver_is_dirty = True. The dirty flag was intended to force a rebuild on the nextfit(), but it also meant the convolver was always rebuilt on the nextfit()call even if nothing had changed — adding unnecessary cost. Worse, iffit()was called very quickly aftercalculate(), the state was confusing.Fix:
calculatenow creates a localconvolvervariable for its own use and passes it directly to_calculate, leavingself._convolvercompletely untouched.8.
Analysis._ensure_analysis_list_currentclearing the dirty flag when Q is NoneWhat broke: The logic was:
If
Qwas not yet set, the dirty flag was cleared without rebuilding. A subsequentanalysis.Q = ...assignment would then not trigger a rebuild because the flag was already clean.Fix: Collapsed to
if self._analysis_list_is_dirty and self.Q is not None: ... self._analysis_list_is_dirty = False, so the flag is only cleared when the rebuild actually runs.Regression test:
test_ensure_analysis_list_current_stays_dirty_when_Q_is_noneintest_analysis.py.9.
Analysis1d._on_Q_index_changedcrashing when Q index was set to NoneWhat broke: During
Analysis1d.__init__,Q_indexis initialised toNone. The_on_Q_index_changedcallback immediately tried to callself.experiment.get_masked_energy(Q_index=None), which raised becauseNoneis not a valid index.Fix: Early return added:
if self._Q_index is None: self._masked_energy = None; self._convolver_is_dirty = True; return.10.
Analysis1d._on_experiment_changedleaving_masked_energystaleWhat broke: Swapping the experiment on an already-initialised
Analysis1d(i.e. afterQ_indexwas set) called the base class handler but did not re-fetch the masked energy for the current Q index. Subsequent calls to_calculateused the energy grid from the old experiment.Fix: A check is added that re-fetches
_masked_energywhen both_Q_indexand the newexperimentare non-None.11.
binned_data['Q', idx]bypassing data masks in residuals and plotsWhat broke:
Analysis1d._create_residuals_arrayanddata_and_model_to_datagroupusedself.experiment.binned_data['Q', self.Q_index]directly, giving the full data slice including NaN / masked points. The model prediction, however, was evaluated only on the finite energy grid (viaget_masked_energy). When NaN points existed, the two arrays had different lengths and the subtraction crashed.Fix: Both sites now call
self.experiment.get_masked_binned_data(Q_index=self.Q_index), which applies the same finite-energy mask as the model path.Regression tests:
test_create_residuals_array_with_nan_data_does_not_crash,test_data_and_model_to_datagroup_with_nan_excludes_nan_from_dataintest_analysis1d.py.test_fit_all_Q_simultaneously_with_nan_dataintest_analysis.py. NewExperimentmethods are covered bytest_get_masked_binned_data_with_data,test_get_finite_energy_mask_with_dataintest_experiment.py.12. Shared
ConvolutionSettingsacross Q indices in multi-QAnalysisWhat broke: All
Analysis1dchild objects created byAnalysis._create_analysis_listreceived the sameConvolutionSettingsinstance.convolution_plan_is_validis a flag on that object. When any one Q slice invalidated the plan (e.g. by updating energy), the flag was cleared for every other Q slice too, forcing unnecessary full plan rebuilds.Fix: Each
Analysis1dnow receives its owncopy(self.convolution_settings)so the validity flag is tracked independently per Q index.13.
NumericalConvolutiondetailed balance factor missing the even-length offsetWhat broke:
NumericalConvolution.convolutionapplied the energy-even-length offset when evaluating the sample model values, but thedetailed_balance_factorwas computed onenergy_dense - energy_offset(without the even-length offset). This meant the detailed balance correction used a slightly shifted energy grid relative to the sample model, introducing a small asymmetry artefact on even-length energy arrays.Fix: The detailed balance factor is now computed on
energy_dense - energy_even_length_offset - offset_value, consistent with how the sample model is evaluated.Regression test:
test_detailed_balance_energy_includes_even_length_offsetintest_numerical_convolution.py.14.
NumericalConvolutionBase.energysetter not updating_energy_gridWhat broke: Setting a new energy array via the
energyproperty setter invalidatedconvolution_plan_is_validbut did not rebuild_energy_grid. Any calculation performed before the next full plan build would use a stale energy grid.Fix:
self._energy_grid = self._create_energy_grid()is called inside the setter.15.
analysis.parameters_to_datasetempty-listnamescheckWhat broke:
if not names:isTruefor bothNoneand an empty list[]. Callingparameters_to_dataset(names=[])was intended to return an empty dataset but instead returned the full dataset (all parameters).Fix: Changed to
if names is None:so an empty list correctly produces an empty output.16.
analysis.parameters_to_datasetpermanently mutating parameters during unit conversionWhat broke: When two parameters with the same name but different units were encountered, the code called
p.convert_unit(target_unit)on the liveParameterobject to normalise units before inserting into the dataset. This permanently changed the unit of the parameter in the model, so subsequent fits or evaluations used the wrong unit.Fix:
p = copy(p)before callingp.convert_unit(...), so the stored parameter is never mutated.