[O2B-1567] Add resync GAQ summary button#2180
Draft
isaachilly wants to merge 122 commits into
Draft
Conversation
Add GaqSummaryRepository and GaqSummaryInvalidationRepository and export them from the repositories index. Update migration to replace the previous invalidated_at field with created_at and add updated_at to mirror default Sequelize tables.
Rename and change mcReproducible to be the coverage float not the boolean.
Forgot to remove.
Add GAQ summary invalidation whenever underlying data affecting GAQ changes. These changes ensure GAQ summary caches/records are marked for recomputation whenever detectors, QC flags, or run QC times that influence GAQ summaries are modified.
Introduce background processing for GAQ summary invalidations and wire it into the app scheduler. Changes include:
- Add gaq config to services config.
- Move GaqService to lib/server/services/gaq and update imports across controllers/use-cases/tests.
- Extend GaqService to pop invalidations and recalculate summaries within a transaction.
- Add GaqWorker that guards concurrent runs and calls GaqService to process a batch of invalid summaries.
- Schedule the GaqWorker in application startup when GAQ recalculation is enabled.
- Add a soft-delete filter (where: { deleted: false }) when querying QC flags to map ids to run numbers as otherwise summary invalidations occur even on already deleted flags.
These changes enable periodic recalculation of GAQ summaries when invalidations are queued in the table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Replace manual aggregation via getGaqCoverages with a direct query to GaqSummary table. getSummary now builds a where clause and formats the summaries, if returned, via a new _formatSummary helper that now the mcReproducibleAsNotBad logic resides in.
This will be useful for when we want to set the cache header.
Add a POST /gaq/recalculate API and corresponding handler to trigger recalculation of GAQ summaries for a data pass and optional run ranges. In GaqService: include invalidation flag on fetched summaries, propagate it to formatted output, ensure upserts update updatedAt, adjust transaction handling, and add recalculateSummaries(dataPassId, runNumbers) which resolves run ranges, enqueues invalidations for matching DataPassRun entries, and returns the number of summaries scheduled for recalculation.
- Add new getGAQSummaryDisplay.js to render GAQ state and per-run recalculate button for admins. - Update getQcSummaryDisplay to accept an options.classes param and render an "invalidated" clock indicator when summary.invalidated is set. - implement recalculateGaqSummary(runNumbers) to POST to /api/qcFlags/summary/gaq/recalculate and refresh summaries.
…nto feature/O2B-1567/Add-resync-GAQ-summary-button
Validate GAQ summary invalidation behaviour on QC flag create/verify/delete, deleteAllForDataPass, explicit/default GAQ detector changes, and run QC time updates.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Converted gaqWorker to a singleton. This allows adding of pause() and resume() methods to prevent the worker from processing invalidated summaries during test execution. Reduced the default GAQ recalculation period from 1 minute to 10 seconds to improve test suite performance.
Adds tests that verify the worker removes an invalidation and adds a summary, upserts for an already present summary, batch processes correctly, and doesn't run concurrent recalculations. Update resetDatabaseContent to pause/resume the GAQ worker to avoid worker failures when the invalidation table is dropped.
GaqService summary recalculation and invalidation processing functions tested directly circumventing worker. Tests verify correct computed fields, summary upsert, behaviour when no coverage exists, and batched invalidation processing.
…into feature/O2B-1567/Add-resync-GAQ-summary-button
Add a calculation_failed boolean column to gaq_summaries. This allows us to know whether a summary has been attempted to be calculated but unsuccessful due to limited data etc.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Remove not-null constraints from several GAQ summary columns in the migration to allow NULL when values are unavailable.
getSummary previously ran arithmetic on raw summary rows without checking for null coverage values. Making "no data yet" look identical to a real 0% coverage. _formatSummary now short-circuits when badRunCoverage is null and returns a full RunGaqSummary shape with null coverage fields. The single-run branch is switched to findOne for a cleaner row-or-null result, and the JSDoc typedef gains the missing undefinedQualityPeriodsCount field. Adds tests covering: single-run, no row, notComputable, invalidated-never-computed, computed-then-invalidated (stale values preserved), data-pass isolation, empty map, full mixed-state map, and the mcReproducibleAsNotBad invariant.
After migrating getSummary to read from the precomputed table, flag creation only invalidates it no longer computes. The background worker eventually fills these in on its 30s tick, but the test runs faster than that, so the assertions saw empty/null data. Call calculateAndStoreGaqSummary for each affected run after creating flags, so the read path has rows to return.
Seeds gaq_summaries with the rows the worker would produce against the existing flag/detector fixtures, so UI tests can assert against GAQ values without waiting on the background recompute.
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
This reverts commit 6b72a6a.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
…nto feature/O2B-1567/Add-resync-GAQ-summary-button
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| getPopoverSelector, | ||
| getInnerText, | ||
| getPopoverInnerText, | ||
| getColumnCellsInnerTexts, |
| const DataPassRepository = require('../../../lib/database/repositories/DataPassRepository.js'); | ||
| const { BkpRoles } = require('../../../lib/domain/enums/BkpRoles.js'); | ||
| const { navigateToRunsPerDataPass } = require('./navigationUtils.js'); | ||
| const { invalid } = require('joi'); |
| * @param {number} ms milliseconds to wait | ||
| * @return {Promise<void>} | ||
| */ | ||
| const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); |
isaachilly
changed the title
…-invalidation-mechanism" This reverts commit 23f6e04.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…into improv/O2B-1565/Migrate-GAQ-Summary-get-to-use-table
…nto feature/O2B-1567/Add-resync-GAQ-summary-button
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2180 +/- ##
===========================================
- Coverage 45.82% 34.96% -10.86%
===========================================
Files 1039 1044 +5
Lines 17189 17390 +201
Branches 3138 3189 +51
===========================================
- Hits 7877 6081 -1796
- Misses 9312 11309 +1997 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have a JIRA ticket
Notable changes for users:
Notable changes for developers:
Changes made to the database: