Skip to content

A solver family's cached data is one object, not eighteen members - #185

Open
BDonnot wants to merge 3 commits into
dev_0.13.2from
claude/solver-side-cache-data-class
Open

A solver family's cached data is one object, not eighteen members#185
BDonnot wants to merge 3 commits into
dev_0.13.2from
claude/solver-side-cache-data-class

Conversation

@BDonnot

@BDonnot BDonnot commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

One commit, fresh from dev_0.13.2. Replaces #184, which took the opposite approach (add checks that the eighteen loose members agree) and is closed.

The problem, concretely

pre_process_solver fills in containers you hand it. Two callers hand it different ones:

// LSGrid::ac_pf -- passes the grid's own members
pre_process_solver(Vinit, acSbus_, Ybus_ac_, id_me_to_ac_solver_, ...);

// BaseBatchSolverSynch::prepare_solver_input_base -- passes the BATCH's members
_grid_model.pre_process_solver(Vinit, Sbus_, Ybus_, id_me_to_solver_, ...);

Six of the nine outputs were parameters. Three were not. At the top of _pre_process_solver_impl:

RealVect & slack_weights = is_ac ? slack_weights_ac_ : slack_weights_dc_;
SolverBusIdVect & bus_pv = is_ac ? bus_pv_ac_ : bus_pv_dc_;
SolverBusIdVect & bus_pq = is_ac ? bus_pq_ac_ : bus_pq_dc_;

Hardcoded to the grid's members whatever the caller passed for the rest. One call, output split across two objects. The tell was already sitting in the batch:

bus_pv_ = _grid_model.get_ac_pv_solver();   // reach back into the grid to collect
bus_pq_ = _grid_model.get_ac_pq_solver();   // the other half of my own output
slack_weights_ = _grid_model.get_ac_slack_weights_solver();
// TODO copies are made here, which is not ideal

Three consequences, all silent:

  1. A batch build wrote its pv-pq split and slack weights into the grid's cache, next to a Ybus built for a different bus labelling — and the grid went on claiming that mixture was reusable. Checked against dev_0.13.2: need_reset_solver() stays false across such a build.
  2. The reuse guard sized one owner's containers against the other's. Sizes agree far more often than labellings do, so a caller reusing its containers with a "nothing changed" control would have skipped fillpv_pq and stamped this grid's split onto its system — converged, plausible, wrong.
  3. _algo.reset() / tell_solver_control() fired on the grid's own algorithm for a solve it would never perform, discarding a factorization its next powerflow would have reused.

Nothing reaches any of this today. The batch works on a private LSGrid copy and always calls tell_all_changed() first. Both are accidents; neither is written down as a requirement, and the obvious next optimisation in there (keep Ybus across scenarios) removes the second one.

The fix: make them one object

src/core/SolverSideCache.hpp, split by type, not lifetime:

contents
SolverBusLayout labelling, slack, pv-pq split, connectivity snapshot, allow_reuse, algo_needs_rebuild — everything whose type doesn't mention the family's scalar
SolverSideCache<T> the above, plus mat and inj. cplx_type → AC (Ybus/Sbus), real_type → DC (Bbus/Pbus)

The split exists so the batch — which runs AC or DC, never both — can name the family-agnostic half: 82 of its 150 call sites don't care which family, and go through active_layout() instead of a ternary at every use.

LSGrid holds ac_cache_ / dc_cache_. pre_process_solver / pre_process_dc_solver take one cache reference. There is no way to hand either function half of one.

What this deletes rather than adds

  • The reuse guard is cache.is_usable(nb_bus) — one place, instead of eight comparisons written inline against a mix of two objects' members.
  • "Is this my cache?" is &c == &ac_cache_, two overloads picked by the family's own type. Solver-side data of a powerflow belongs to exactly one owner #184 needed nine pointer comparisons and a throw on a partial match; that whole function is gone.
  • _mark_cache_valid / prevent_*_cache_reuse / init_bus_status lose their if(ac) x_ac_ else x_dc_ bodies.
  • BaseBatchSolverSynch's eleven loose members become its own two caches; the read-back above, and its three copies, are gone.

It's also the extension point: remote / shared voltage-control layouts, HVDC droop data — add a field and it inherits the lifetime, invalidation and consistency rules of everything already there.

DualAlgoControl deliberately stays put. It's threaded through every element-container mutator signature and is already correctly per-family.

Behaviour changes (all in the foreign-build path)

  • A build into a caller's cache never reuses. solver_control, the snapshot init_bus_status() raises its flags against, and those flags all describe this grid and say nothing about someone else's cache.
  • It retires this grid's cache for that family afterwards. The labelling and the split still have to be published — the NR extensions read them back through lsgrid_ptr rather than from what the solver was handed, including bus_pq, which fill_voltage_control_solver_data needs and which the old write-through supplied by accident. Publishing the matrix too would mean copying it, so what remains is a view for the extensions, not a cache: snapshot cleared, control raised, next own powerflow rebuilds.
  • The grid's own algorithm is no longer reset or reconfigured for a solve it will never perform.

⚠️ ABI

This changes LSGrid's member layout. The note that sat on _forced_ref_slack_bus_id ("declared LAST so existing member offsets are unchanged — ABI-stable for the gpusim2grid cross-module LSGrid cast") cannot survive regrouping scattered members. docs/solver_plugin.rst already requires the same version of headers at runtime ("different BaseAlgo layout ... undefined behaviour"), and 1.0.0 already carries several [BREAKING] entries — so this is flagged in the changelog rather than worked around. gpusim2grid must be rebuilt against these headers.

Cost — measured

Callgrind slope, 200→1200 powerflows on the exotic-elements IEEE14 grid (so construction cancels):

dev_0.13.2 this branch delta
AC powerflow 839 340 instr/pf 839 541 +200 (+0.024 %)
DC, warm cache 28 095 instr/pf 28 309 +214 (+0.76 %)

The DC row is the floor — the cheapest powerflow this library can run.

Roughly two thirds of that is not this change: a per-function diff of the two profiles puts +74/pf in compute_results_tsc_rxha_no_amps and +60/pf in an Eigen::Ref helper, neither of which the diff touches — inlining decisions that moved when the translation unit was recompiled. Wall clock shows nothing: run-to-run spread on one unchanged binary (2464 → 2744 ns on the same DC case) exceeds the gap between the two binaries.

Tests

  • test_cache_reuse.cpp, "the structural half … unset_changes" — the existing [unset_changes] sections all reach it through allow_cache_reuse(false), i.e. both families off, so the family that runs is stopped by is_usable's first line (if(!allow_reuse)) and those sections pass with the rest of is_usable deleted. Leaving one family automatic is what exercises the structural checks. Verified: gutting is_usable to return allow_reuse; makes it SIGSEGV.
  • test_cache_reuse.cpp, "a build into someone else's cache …" — a foreign build fills the caller's cache completely, publishes what the extensions read, and retires the grid's. The need_reset_solver() assertion is the one that separates this from dev_0.13.2.
  • test_batch_voltage_control.cpp — the source grid solves identically on both sides of a TimeSeries and a ContingencyAnalysis.

210 test cases / 5815 assertions pass under Release, C++14 (LS2G_CXX_STANDARD=14), ASan+UBSan, and valgrind (0 errors, no leaks).

The python layer could not be exercised locally (no pybind11 / numpy on the machine this ran on) — CircleCI's test_legacy_* jobs cover it. No python-visible API changed; pre_process_solver was never bound, and every &LSGrid:: symbol the bindings reference still resolves.


🤖 Generated with Claude Code

https://claude.ai/code/session_01UtvkqXMvgCig75gruZCs7u


Generated by Claude Code

…embers

The bus labelling, Ybus / Sbus, the slack, the pv-pq split and the connectivity
snapshot are one picture of the grid taken at one instant, all of it expressed in
ONE bus labelling. They were eighteen separate LSGrid members, and
`pre_process_solver` took six of them as parameters while reaching for the other
three itself -- so a caller building into its own vectors got six of its outputs
and the grid got three of them.

That is a real defect in the batch path. A TimeSeries / ContingencyAnalysis prep
wrote its pv-pq split and its slack weights into the grid's cache, next to a Ybus
built for a different labelling, and left that grid still claiming the mixture was
reusable (checked against dev_0.13.2: need_reset_solver() stays false across such
a build). The reuse guard then sized one owner's containers against the other's;
sizes agree far more often than labellings do, so a caller reusing its containers
with a "nothing changed" control would have skipped fillpv_pq and stamped this
grid's split onto its own system -- converged, plausible, wrong. Nothing reaches
it today: the batch works on a private copy and always asks for a full rebuild.
Both of those are accidents, and neither is written down as a requirement.

So rather than adding a check that the nine parts agree, make them one object.

  src/core/SolverSideCache.hpp
    SolverBusLayout      -- labelling, slack, pv-pq split, connectivity snapshot,
                            allow_reuse, algo_needs_rebuild (types that do not
                            mention the family's scalar)
    SolverSideCache<T>   -- the above, plus mat and inj (the two that do)
                            T = cplx_type -> AC (Ybus / Sbus)
                            T = real_type -> DC (Bbus / Pbus)

  LSGrid holds ac_cache_ / dc_cache_. pre_process_solver / pre_process_dc_solver
  take one cache reference. There is no way to hand either half of one.

What this deletes, rather than adds:
- the reuse guard is `cache.is_usable(nb_bus)`, in one place, instead of eight
  comparisons written inline against a mix of two objects' members;
- "is this my cache?" is `&c == &ac_cache_`, two overloads picked by the family's
  own type, instead of counting nine pointer comparisons and throwing on a
  partial match;
- `_mark_cache_valid` / `prevent_*_cache_reuse` / `init_bus_status` lose their
  `if(ac) x_ac_ else x_dc_` bodies;
- BaseBatchSolverSynch's eleven loose members become its own two caches, and the
  read-back of _grid_model.get_ac_pv_solver() -- with the three copies its own
  `TODO copies are made here, which is not ideal` flagged -- is gone: one call
  fills the whole thing, into vectors the batch owns.

And it is the extension point that was asked for: remote / shared voltage-control
layouts, HVDC droop data, whatever comes next, go in as another field and inherit
the lifetime, invalidation and consistency rules of everything already there.
`algo_controler_` (DualAlgoControl) deliberately stays where it is -- it is
threaded through every element container's mutator signature and is already
correctly per family.

Two behaviours change, both in the foreign-build path:
- a build into a caller's cache never reuses (`solver_control`, the snapshot
  init_bus_status() compares against and the flags it raises all describe THIS
  grid, and say nothing about someone else's cache);
- it retires this grid's cache for that family afterwards. The labelling and the
  split still have to be published, because the NR extensions read them back
  through `lsgrid_ptr` rather than from what the solver was handed -- including
  bus_pq, which fill_voltage_control_solver_data needs and which the old
  write-through supplied by accident. Publishing the matrix too would mean copying
  it, so what is left is a view for the extensions, not a cache: the snapshot is
  cleared and the control raised, so this grid's next own powerflow rebuilds.
- the grid's own algorithm is no longer reset / reconfigured for a solve it will
  never perform.

ABI: this changes LSGrid's member layout, so a consumer that casts an LSGrid
across a module boundary (gpusim2grid -- see the note that used to sit on
_forced_ref_slack_bus_id) must be rebuilt against these headers. That is already
what docs/solver_plugin.rst requires of plugins ("the same version of
lightsim2grid headers that is installed at runtime ... different BaseAlgo
layout"). Flagged in the changelog.

Cost, callgrind instruction counts (slope between 200 and 1200 powerflows on the
exotic-elements IEEE14 grid, so construction cancels): +200 instr on an AC
powerflow (839 340 -> 839 541, +0.024%) and +214 on a warm-cache DC one
(28 095 -> 28 309, +0.76%, and that is the cheapest powerflow this library can
run). Roughly two thirds of it is not this change: a per-function diff of the
profiles puts +74/pf in compute_results_tsc_rxha_no_amps and +60/pf in an
Eigen::Ref helper, neither of which the diff touches -- inlining decisions that
moved when the translation unit was recompiled. Wall clock shows nothing: the
run-to-run spread on one unchanged binary (2464 -> 2744 ns on the same DC case)
is larger than the gap between the two.

Tests:
- test_cache_reuse.cpp, "the structural half ... unset_changes": the existing
  [unset_changes] sections all reach unset_changes() through
  allow_cache_reuse(false), ie BOTH families off, so the family that runs is
  stopped by is_usable's first line (`if(!allow_reuse)`) and those sections pass
  with the rest of is_usable deleted. Leaving ONE family automatic is what
  exercises it; verified to SIGSEGV against a build with is_usable gutted to
  `return allow_reuse;`.
- test_cache_reuse.cpp, "a build into someone else's cache ...": a foreign build
  fills the caller's cache completely, publishes what the extensions read, and
  retires the grid's -- so the grid's own next powerflow is unaffected. The
  need_reset_solver() assertion is the one that separates this from dev_0.13.2.
- test_batch_voltage_control.cpp: the source grid solves identically on both sides
  of a TimeSeries and a ContingencyAnalysis.

Verified: 210 test cases / 5815 assertions pass under Release, under C++14
(LS2G_CXX_STANDARD=14), under ASan+UBSan, and under valgrind (0 errors, no
leaks). The python layer could not be exercised here (no pybind11 / numpy on this
machine); no python-visible API changes.

Signed-off-by: DONNOT Benjamin <benjamin.donnot@rte-france.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtvkqXMvgCig75gruZCs7u
BDonnot and others added 2 commits August 23, 2026 14:20
init_bus_status() took SubstationContainer::get_bus_status() by value:

    const std::vector<bool> new_status = substations_.get_bus_status();

The accessor already returns `const std::vector<bool> &`, so this copied the
whole vector into a local, to be read twice by _flag_dimension_change and
thrown away. Every mutation of the bus status happens in the disconnect /
reconnect calls above the line, and _flag_dimension_change only reads, so a
reference is safe.

One missing `&`. O(nb_bus) per powerflow that reaches init_bus_status, which is
every powerflow that rebuilds and every one where an element changed bus.

Verified: 210 test cases / 5815 assertions pass under Release.

Signed-off-by: DONNOT Benjamin <benjamin.donnot@rte-france.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtvkqXMvgCig75gruZCs7u
Same split as compute_pf_with_input_validation / compute_pf: the user-facing
entry point does not take the caller's word for it, the internal one does,
because it has just built the data itself.

  unset_changes()            user-facing, BOTH families, checked
  _mark_cache_valid(...)     internal, one family, no checks

`unset_changes()` still marks both families -- that is its historical contract
and pre-1.0.0 code relies on it -- but each is now verified before the claim is
recorded, and a family that cannot back it is retired rather than marked.

The check that matters is not the one I expected
------------------------------------------------
The obvious check is "does the cache hold a system of the right shape". It is
necessary -- that is what stands between a claim about a family that never
solved and an out-of-bounds read -- but it is nowhere near sufficient, and the
new test caught it: after

    ac_pf(); deactivate_powerline(4); unset_changes(); ac_pf();

the answer was wrong by 0.038 pu. Deactivating that line changes no bus's
connected/disconnected status and no vector's size, so every structural check
passes -- and `tell_none_changed()` then cleared the pending `recompute_ybus`,
so the next powerflow re-solved a Ybus that still contained the line.

A cache cannot tell from its own contents that it is stale. Change an
impedance, a tap, a load target: every size and every bus status is exactly
what it was. What knows is the element containers, which declare every change
through AlgoControl in their own modifiers -- so the load-bearing test is to
READ that back, via the new `AlgoControl::nothing_changed()` (the exact negation
of `tell_all_changed()`). The containers stay the sole authority: nothing here
second-guesses what they declared, adds flags of its own, or recomputes
connectivity behind them.

The powerflow path stops re-checking
-------------------------------------
`_pre_process_solver_impl` now asks only the switch (`!own_cache ||
!cache.may_be_reused()`). It can, because every other way the flags reach it can
only make a cache MORE stale, never falsely fresh: the containers raise them as
the grid is modified, AlgoControl's constructor asks for a full rebuild,
set_state / the copy ctor / a divergence reset, and python cannot clear them
(`get_*_algo_controler()` is bound read-only -- binding_misc.cpp exposes the
has_* / need_* getters and no tell_*). The two entry points that CAN claim
"nothing changed" without having built anything now verify it themselves:
`unset_changes()`, and `check_solution()`, whose weaker
`id_me_to_solver.size() > 0` guard is replaced by the same consistency check.

A debug-only assertion keeps that reasoning honest: free under -DNDEBUG (what
the wheels ship), and it fires in the C++ suite -- which CI runs under ASan,
UBSan and valgrind -- the day a third claimant appears. Verified non-vacuous:
making `unset_changes()` mark unconditionally aborts the Debug build.

`SolverSideCache`'s predicates are now two separate questions, which is what
made the above possible to reason about at all:
  is_consistent(nb_bus)   is the data there and self-consistent (sizes only)
  may_be_reused()         is reuse allowed at all
Neither asks "has the grid changed" -- that is not a question a cache can answer
about itself, and AlgoControl already answers it.

Tests: test_cache_reuse.cpp's [unset_changes] case rewritten. It used to pin the
dangerous intermediate state -- that a never-built family got marked "in sync",
with the powerflow path catching it later. It now pins the opposite: the claim
is refused at the point it is made. Sections cover a family that never solved,
each family solved alone, a pending topology change (the 0.038 pu case), a
genuinely valid cache claimed repeatedly, and the three sequences that used to
segfault. Verified the flag test bites: removing it reproduces the wrong answer.

Verified: 210 test cases / 5830 assertions pass under Release, C++14, Debug
(assertions live), ASan+UBSan, and valgrind (0 errors, no leaks). The python
layer could not be exercised here (no pybind11 / numpy on this machine).

Signed-off-by: DONNOT Benjamin <benjamin.donnot@rte-france.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtvkqXMvgCig75gruZCs7u
@BDonnot
BDonnot force-pushed the claude/solver-side-cache-data-class branch from 73ca033 to 03260b9 Compare August 23, 2026 15:54
@BDonnot BDonnot closed this Aug 23, 2026
@BDonnot BDonnot reopened this Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant