fix: preserve scraping metadata - #2254

Open

vegettto wants to merge 1 commit into

unclecode:mainfrom

vegettto:fix/scraping-metadata

vegettto commented Sep 13, 2026 •

edited

Loading

Summary

Fix two small scraping metadata issues:

Preserve the root <html lang> attribute when lxml parses a complete HTML document.
Return successful image Content-Length values instead of discarding them.

List of files changed and why

crawl4ai/async_url_seeder.py - Preserve the language declaration on the root HTML element.
crawl4ai/content_scraping_strategy.py - Remove a finally: return that always discarded successful image-size results.
tests/unit/test_sitemap_namespace_parsing.py - Add regression coverage for root-level HTML language extraction.
tests/unit/test_content_scraping_helpers.py - Add regression coverage for successful image-size extraction.

How Has This Been Tested?

Verified both corrected behaviors with focused local checks.
Compiled the modified Python modules successfully.
The full pytest suite was not run locally because the project test dependencies were unavailable in the local environment.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added/updated unit tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes


          fix: preserve scraping metadata

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet