[O2B-1565] Migrate GAQ summary to use table#2179
Draft
isaachilly wants to merge 92 commits into
Draft
Conversation
Add GaqSummaryRepository and GaqSummaryInvalidationRepository and export them from the repositories index. Update migration to replace the previous invalidated_at field with created_at and add updated_at to mirror default Sequelize tables.
Rename and change mcReproducible to be the coverage float not the boolean.
Forgot to remove.
Add GAQ summary invalidation whenever underlying data affecting GAQ changes. These changes ensure GAQ summary caches/records are marked for recomputation whenever detectors, QC flags, or run QC times that influence GAQ summaries are modified.
Introduce background processing for GAQ summary invalidations and wire it into the app scheduler. Changes include:
- Add gaq config to services config.
- Move GaqService to lib/server/services/gaq and update imports across controllers/use-cases/tests.
- Extend GaqService to pop invalidations and recalculate summaries within a transaction.
- Add GaqWorker that guards concurrent runs and calls GaqService to process a batch of invalid summaries.
- Schedule the GaqWorker in application startup when GAQ recalculation is enabled.
- Add a soft-delete filter (where: { deleted: false }) when querying QC flags to map ids to run numbers as otherwise summary invalidations occur even on already deleted flags.
These changes enable periodic recalculation of GAQ summaries when invalidations are queued in the table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Replace manual aggregation via getGaqCoverages with a direct query to GaqSummary table. getSummary now builds a where clause and formats the summaries, if returned, via a new _formatSummary helper that now the mcReproducibleAsNotBad logic resides in.
Validate GAQ summary invalidation behaviour on QC flag create/verify/delete, deleteAllForDataPass, explicit/default GAQ detector changes, and run QC time updates.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Converted gaqWorker to a singleton. This allows adding of pause() and resume() methods to prevent the worker from processing invalidated summaries during test execution. Reduced the default GAQ recalculation period from 1 minute to 10 seconds to improve test suite performance.
Adds tests that verify the worker removes an invalidation and adds a summary, upserts for an already present summary, batch processes correctly, and doesn't run concurrent recalculations. Update resetDatabaseContent to pause/resume the GAQ worker to avoid worker failures when the invalidation table is dropped.
GaqService summary recalculation and invalidation processing functions tested directly circumventing worker. Tests verify correct computed fields, summary upsert, behaviour when no coverage exists, and batched invalidation processing.
Add a calculation_failed boolean column to gaq_summaries. This allows us to know whether a summary has been attempted to be calculated but unsuccessful due to limited data etc.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Remove not-null constraints from several GAQ summary columns in the migration to allow NULL when values are unavailable.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Stop returning early when summary is missing; always upsert a GaqSummary record and include a calculationFailed boolean. This ensures failed calculations are persisted for monitoring instead of being silently skipped.
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
Update test to expect that when there is no coverage data for a run, a GAQ summary is stored with calculationFailed set to true.
resetDatabaseContent unconditionally called gaqWorker.resume(), which silently undid any pause set up by a calling test suite. Capture the worker's paused state before pausing and only resume it if it wasn't already paused on entry. Adds an isPaused getter on GaqWorker to expose the state without reaching into the private field.
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
getSummary previously ran arithmetic on raw summary rows without checking for null coverage values. Making "no data yet" look identical to a real 0% coverage. _formatSummary now short-circuits when badRunCoverage is null and returns a full RunGaqSummary shape with null coverage fields. The single-run branch is switched to findOne for a cleaner row-or-null result, and the JSDoc typedef gains the missing undefinedQualityPeriodsCount field. Adds tests covering: single-run, no row, notComputable, invalidated-never-computed, computed-then-invalidated (stale values preserved), data-pass isolation, empty map, full mixed-state map, and the mcReproducibleAsNotBad invariant.
After migrating getSummary to read from the precomputed table, flag creation only invalidates it no longer computes. The background worker eventually fills these in on its 30s tick, but the test runs faster than that, so the assertions saw empty/null data. Call calculateAndStoreGaqSummary for each affected run after creating flags, so the read path has rows to return.
Seeds gaq_summaries with the rows the worker would produce against the existing flag/detector fixtures, so UI tests can assert against GAQ values without waiting on the background recompute.
isaachilly
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2179 +/- ##
==========================================
+ Coverage 45.82% 46.51% +0.69%
==========================================
Files 1039 1043 +4
Lines 17189 17323 +134
Branches 3138 3170 +32
==========================================
+ Hits 7877 8058 +181
+ Misses 9312 9265 -47 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
This reverts commit 6b72a6a.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
…-invalidation-mechanism" This reverts commit 23f6e04.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have a JIRA ticket
Notable changes for users:
Notable changes for developers:
Changes made to the database: