Update Maven HTML listings from immediate directory children

[gerrod3]

A small Maven repository modification currently queries descendant content to regenerate ancestor HTML pages. This adds an independent incremental_index_pages=true repository label: maintain each directory's immediate children, render dirty pages, and reuse unchanged page artifacts and memberships.

Extracted from #498 and based directly on main. It has no dependency on the experimental binary-index integration or S3 index storage. It retains ordinary MavenIndexPage content and the existing content-serving behavior.

• A derived version cursor detects failed versions and gaps, forcing a rebuild before incremental updates resume. No-op versions preserve the cursor.
• Creating/removing directories updates parent links; changing a file normally changes only its containing page. Directory links omit aggregate size/date, while file links retain size and membership date.
• Adds a migration for the summary tables. Bootstrap and explicit repair can scan the whole repository; one very large directory still needs memory proportional to its listing.

Review the two commits in order: generator/schema, then lifecycle coverage and documentation. A review-order comment identifies the key correctness checks.

Validation: full unit suite 181 passed, 3 optional S3 tests skipped, 4 subtests passed; eight directory lifecycle tests cover a public modify task with 200 orphan uploads, retention, no-ops, failed versions, repair, opt-in gaps, and base-version switching. Ruff, public pulpcore import checks, and migration consistency pass. A combined checkout with #498 also passed all 24 lifecycle tests with both opt-ins enabled. Production-size throughput is not established by these tests.

AI disclosure: code, tests, documentation and this description were generated with Codex. Commits carry Generated-by: Codex; human review is pending.

[gerrod3]

Suggested review order:

1. models.py and 0015_incremental_index_pages.py: check the independent opt-in, summary ownership, directory/child uniqueness, and partial dirty-directory index.
2. directory_pages.py: follow the previous-version cursor check, changed-path replacement, bottom-up empty-directory pruning, and SHA-based page reuse. A failed finalizer must cause the next attempt to rebuild; a no-op must retain the previous cursor.
3. test_directory_pages.py: start with ancestor-page reuse and the 200-orphan public modify test, then review failure recovery, retention, explicit repair and base-version switching.
4. docs/user/guides/index-pages.md: review the deliberate removal of aggregate dates/sizes from directory links and the limits of bootstrap/very large listings.

The binary-index integration is separate in #498. Neither PR requires the other, and HTML pages remain normal content served by the existing handler. Review human approval is still needed for this AI-generated implementation.