Hold back HTML tables above a row limit and summarise oversized index tabs - #227
Merged
Merged
Conversation
… tabs Every row of every artifact went into its HTML page, and the table script then read them all back into memory. Measured in Chromium on a 16-column messages page: 25,000 rows were usable 5.9 s after opening, 50,000 in 11.3 s and 100,000 in 25.8 s with the browser holding 1.7 GB for the page; at 150,000 the DataTables call failed with "Maximum call stack size exceeded" inside jQuery, the loading spinner never cleared and the browser held 9 GB. A 1.2 million row page was 415 MB. The index copied the run log and the processed files list into two tabs: a 43 MB list of 229,869 paths opened in under a second, but clicking its tab laid out 230,000 list items, blocked the page for over a minute and took the browser to 12 GB. write_artifact_data_table now writes a notice instead of a table above HTML_TABLE_ROW_LIMIT (50,000): the row count, the limit, and where the complete rows are, the LAVA database and the TSV export. TSV, timeline, KML and LAVA output are unchanged, the page keeps its place in the side bar, and the index Details tab lists every held-back table with a link. A new --html_row_limit argument changes the limit; 0 writes every table. Above 2 MB the run log tab keeps 200 lines from each end and the processed files tab keeps each artifact's patterns with their counts and first 200 paths, both linking to the full file. The LAVA-only notices no longer describe LAVA as unreleased. A 60,000-row store now writes a 6 KB page in place of a 21 MB one with the same TSV and LAVA rows; a 25,000-row store's page, TSV and LAVA tables are byte-identical to before. On a full run of a public iOS 17 image eight tables were above the limit, the largest a 6.65 million row FSEvents page of 2.1 GB; the HTML folder went from 5.2 GB to 1.6 GB and the index from 16.2 MB to 3.9 MB with every LAVA table and TSV the same. Tests cover the notice, the complete TSV and LAVA rows behind it, the index entry, and the summarised tabs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Holds back HTML tables above a row limit and summarises oversized index tabs.
--html_row_limitchanges the limit;0writes every table.Same change in all five cores.
🤖 Generated with Claude Code