Conversation
`extract_table_data` read only `.//td` for body rows, so a `<th scope="row">` was dropped: every other cell shifted one place left and the row was padded with "" on the right. For a documentation table whose first column is the product or parameter name, that column disappeared from `result.tables` altogether. `rowspan` was not handled at all, so the rows a spanning cell covers shifted left by one and stopped lining up with the header. The value is now carried into each row the cell declares. Including `th` means the first row is a body row too when there is no `thead`. It was skipped before only because it held no `td`, so it is now skipped by identity against the row the headers came from. `colspan` and `rowspan` come from the document and can be absent, empty, zero, negative or not a number. `int()` raised on the last of those and zero would have dropped the cell, so both are read through one helper that means "no span" for all of them, the way a browser does. Fixes unclecode#2258
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
DefaultTableExtraction.extract_table_dataread only.//tdfor body rows, so a<th scope="row">was dropped — the remaining cells shifted one place left and the row was padded with""on the right.rowspanwas not handled at all, so the rows a spanning cell covers shifted left too.Run against the issue's own markup, before:
after:
For a documentation table whose first column is a product or parameter name, that column disappeared from
result.tablesentirely. This is the follow-up to #2007: since v0.9.1 the spans survive incleaned_html, and the extractor now uses them.Two things the change had to handle that the issue does not mention
The header row becomes a body row. Without a
theadthe first row supplies the headers, and it is also in.//tr[not(ancestor::thead)]. It used to be skipped by accident — it holds notd, sorow_datacame out empty. Now thatthcounts, it has to be skipped on purpose, by identity against the row the headers came from.test_the_header_row_is_not_repeated_as_datapins that, and it fails if the skip is removed.A span attribute is whatever the document said.
int(cell.get("colspan", 1))raisesValueErroroncolspan="nonsense"— a cell my own test wrote before I noticed — andcolspan="0"would have dropped the cell and shifted the row, which is the same misalignment this change exists to fix. Both spans are now read through one helper that treats absent, empty, zero, negative and non-numeric alike as "no span", which is what a browser does.Tests
tests/test_table_gfm_compliance.py(the neighbouring table tests): 13 passed, unchanged.Against the extractor on
main, with the new file in place, 5 fail — the two shapes from the issue and three span combinations:Each cell was checked against a mutant that breaks the thing it names. Baseline 16/16 green, source restored and re-verified; every mutation asserts it changed the file first:
tdin the body (the behaviour onmain)0instead of1The fifth row is why there is a
colspanversion of the bad-span cell as well as arowspanone. With only therowspancases that mutant survived:rowspanof0and of1both mean "no carry", so nothing observable changed. It takes a zerocolspanto see it, and that one drops a cell.The control cell (
test_a_plain_table_is_unchanged) asserts headers, rows and the metadata counts for an ordinary table, so a change that fixed the spans by breaking the common case would not pass.Fixes #2258