Add ArtifactResult, a streaming result path for large artifacts - #234
Merged
Merged
Conversation
Levels iLEAPP #1800 into VLEAPP. A module may return an ArtifactResult from context.create_artifact_result() and add rows one at a time; the rows are written into the LAVA database in batches while the module runs and replayed from there for the HTML, TSV, timeline and KML outputs. Every existing artifact keeps returning its (headers, data_list, source_path) tuple and is unchanged. The batched insert applies to the list path too, which no longer builds a second full list before executemany. Carried over from the iLEAPP work: a module that raises after streaming rows leaves no table and no manifest entry behind; replayed rows keep the values the module wrote (floats, booleans and integers in untyped columns round-trip); a media cell holding several references round-trips as a list; a streamed table above the HTML row limit is held back before any row is read out of LAVA. Storage conventions are this core's own: an ISO string and a datetime object are both stored as a float epoch, as VLEAPP has always done, so a list-returning artifact's LAVA values do not change. VLEAPP's artifact_processor wrapper takes four arguments and is otherwise the iLEAPP function. Tests cover the writer, the replay for each value type, the list path's unchanged storage, and a raising module through artifact_processor for both module styles. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ArtifactResult, the streaming result path from iLEAPP #1800, to VLEAPP.
A module may return a result from context.create_artifact_result() and add rows one at a time; rows go into the LAVA database in batches while the module runs and are replayed from there for the HTML, TSV, timeline and KML outputs. Every existing artifact keeps returning its tuple and is unchanged. The batched insert also applies to the list path, which no longer builds a second full list before executemany.
Carried over: a module that raises after streaming rows leaves no table and no manifest entry; replayed rows keep the values the module wrote; a media cell holding several references round-trips as a list; a streamed table above the HTML row limit is held back before any row is read out of LAVA.
Storage stays this core's own: an ISO string and a datetime object are both stored as a float epoch. The four-argument artifact_processor wrapper is kept.
Verified against main on a real image, list-returning artifacts only, name-keyed: ford_syncg4_logical, 6-artifact profile: 6 of 6 LAVA tables identical, 6 of 6 TSVs identical, 502 timeline rows identical, KML identical. Unit suite and lint_changed pass.
🤖 Generated with Claude Code