Add alert generation code (v01.00 schema) - #7
Open
AshishMahabal wants to merge 1 commit into
Open
Conversation
rusholme
added a commit
that referenced
this pull request
Aug 6, 2026
…defects The external implementation review sampled the extracted stage bodies against the deleted monolith and found five fidelity defects. It sampled; this sweeps. The authority throughout is `5664024^:pipeline/awsBatchSubmitJobs_runSingle*`. #7 SFFT could not be invoked at all. The argv carried six positional inputs and `--sci_star_list`/`--ref_star_list`/`--crossconv_flag`; the tool's own parser takes two positionals and `--scicat`/`--refcat`/`--crossconv` (store_true). Every SFFT-enabled science job died in argparse with status 2. The branch was also inverted: the monolith passes catalogues in the non-rimtimsim case, with 50.0/100.0 bright-source masking, and none in the rimtimsim case, with 20000.0/30.0. Restored, with the crossconv-dependent output filenames the tool actually writes. #8 Statistics lookups named keys that do not exist. `clippedmed` (x4), `datascale`, `gmin`, `gmax`, `npixsat`, `npixnan` — the helper returns `gmed`, `gsigma`, `gdatamin`, `gdatamax`, `satcount`, `nancount`. Every dedicated reference-image job died here. #19 SFFT and naive silently used ZOGY's science. Both detected on ZOGY's Scorr image, weighted with ZOGY's uncertainty image and fitted with ZOGY's difference PSF. Each variant now builds its own uncertainty image, uses its own PSF (the reference PSF for naive, per the monolith's TODO), and detects on the right image. The naive branch had also dropped its coverage-map masking entirely. #20 Positive and negative PSF catalogues overwrote each other — one configured filename for both signs, so both product names carried negative bytes. The schema was also reduced: no RA/Dec columns, no finder catalogue, and the parquet written from the unjoined table. Restored, join included. #21 The clipped science average was computed after fake-source injection, so injected pixels entered the uncertainty model. The monolith computed it at line 798 and injection opened at 806. Split into its own stage, sequenced ahead of injection — a sequence position is visible where an inline computation's is not. #22 The inline reference build dropped `generatePhotUtilsReferenceImageCatalog`. A science job building its own reference produced no reference PSF/finder catalogues. Also restored its saturation-rate divisor: the science monolith divides by the science exposure time, where the dedicated reference-image pipeline divides by 60.0. Both reproduced against their own authority. #23 The difference image was stamped with the reference pipeline's PPID. The monolith read one science ppid (15) and stamped it into both headers. `ppid_for(JOB_TYPE_SCIENCE)` is that value and needs no manifest fact. Two defects the sweep found beyond the review's sample: * Thirteen release-content keys the W4B config migration dropped while stage code still read them by name — the psfcat filename triples for all three variants and the reference image, ZOGY's three outputs, awaicgen's three, bkgest's two, and the naive difference's. Each a KeyError in a stage that had already done real work. * `prepare_zogy_inputs` clipped ZOGY's input reference statistics at the saturation *rate* where the monolith used the undivided level, and fell back to a config key that does not exist when the reference was downloaded rather than built. The regression tests assert against real artefacts, not restated expectations: the SFFT argv is parsed by the tool's own parser, lifted from its source; the statistics key names are read out of the helper's own source; every release key a stage reads is checked against the real TOML. Both were wrong together before, which is what a test of the code against itself cannot catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdQSan34f2B5fg4J8hauKi
rusholme
added a commit
that referenced
this pull request
Aug 6, 2026
…s publishes Two findings, one seam: what a terminal record carries and who is entitled to be registered from one. #2. context.provenance started EMPTY and the entrypoint seeded exactly three keys -- release_content_digest, tessellation_version, tessellation_digest. UnitFacts were never copied in, and `sca` lives on the UNIT rather than on the facts, so it was not even reachable from there. Every fact the registrar asks of science_provenance about WHICH piece of sky an attempt was -- field, fid, rid, sca, the reference identity, the ten sky-position numbers -- was absent from every record production could author. The registrar was not merely untested against production records: no production record could ever have satisfied it, and the first real candidate would have failed forever. hp6/hp9 were worse than absent -- they existed NOWHERE in the pipeline, not in UnitFacts, not in a stage, not in gathering. Derived now from the science image's own centre, transcribed from loadPSFCatIntoDBSourcesTable.py, which is the file that actually populated the columns they are registered into. Not a free choice: an index at a different nside, or in the ring scheme, or with lat/lon swapped, is a valid integer silently naming the wrong patch of sky. The metric names are fixed at the RECORDING sites rather than translated at read time. The live schema settled it -- diffimmeta's real columns are nsexcatsources, scalefacref, dxrmsfin, dyrmsfin, dxmedianfin, dymedianfin, so the registrar speaks the operations schema's own vocabulary and a translation layer would put that vocabulary in a second home. scalefacref is the INVERTED quantity: traced through the legacy registrar, which does no arithmetic and passes the ZOGY config's own key, back to the science monolith that wrote 1./scalefac. Both values are plausible positive floats, so getting it backwards would have silently disagreed with every row already in the column. Registrar tests are rebuilt through build_terminal_record with real StageContext/ProcessingUnit/UnitFacts. Hand-constructed bodies are what let a registrar that no production record could satisfy look tested. #7. post_process called the legacy boolean-returning uploader and discarded the result -- which also silently skips a file that does not exist -- and never called publish_products or context.publish. So published_products stayed empty, the record carried no products key, and a failed upload still closed success/published with an empty terminal product set. Its FITS S3OBJPRF also named job_type/unit while the bytes went to the run/attempt prefix, so the header pointed at a prefix nothing was written to. Candidate selection could not filter by job type -- the attempts table has no job_type column, so that had to be a record-body gate rather than a SQL predicate. Registration jobs closed themselves success/published, which is the SOLE registering pair, so every successful registration pass became an unsupported candidate on the next one. The disposition is derived from what was actually published now, and unregistrable types advance the watermark instead of raising. healpy joins requirements.txt. It has always been in the Dockerfiles and never declared, so "registration works" depended on a fact no dependency list stated -- and with hp6/hp9 NOT NULL, an environment without it cannot register at all. Verified 1.20.0 in the deployed image rather than assumed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alert generation with optional truth tables from OU24.
Will also work with similar structure as RAPID currently generates.