Skip to content

Upgrade to Python 3.11 - #39

Draft
olmos-keepsafe wants to merge 10 commits into
masterfrom
python311-upgrade
Draft

Upgrade to Python 3.11#39
olmos-keepsafe wants to merge 10 commits into
masterfrom
python311-upgrade

Conversation

@olmos-keepsafe

@olmos-keepsafe olmos-keepsafe commented May 12, 2026

Copy link
Copy Markdown

Key changes

  • Migrates packaging from setup.py / setup.cfg to pyproject.toml, sets Python policy to 3.11, and bumps the package version to 1.0.0.
  • Hard-pins runtime/dev dependencies and upgrades Python 3.11 blockers. The final runtime targets include the team-required aiohttp==3.13.2, beautifulsoup4==4.15.0, lxml==6.0.2, Markdown==3.10.2, parse==1.22.1, and KeepSafe sdiff==1.1.0 from immutable html-structure-diff commit 7cac22038c90708296789197d8492d8aa16087be; coverage is refreshed to 7.15.2.
  • Confirms msgpack is not a content-validator dependency or source/test concern, so libks msgpack compatibility wrappers are not applicable here.
  • Replaces nose with pynose, moves Flake8 config into pyproject.toml, and keeps sample-style PYNOSE_SHARED_FLAGS plus CI coverage/xUnit artifact flags.
  • Adds fixture-backed golden compatibility tests for markdown diff shape, Java placeholders, and URL extraction.
  • Adds a minimal content-validator CLI help/version smoke surface because package metadata already declared the console script.
  • Updates Makefile, README, Travis, requirements, git hook wiring, and a repo-specific migration contract.
  • Adds a native CircleCI 2.1 config using reusable executors and commands, with prepare_cache, lint, and test jobs on Python 3.11.13, schema-correct restore/save cache steps, fallback cache keys, test/coverage artifacts, and non-fatal Codecov upload.
  • Aligns CI installation with email-service: load the shared KeepSafe SSH key fingerprint, install committed development requirements from the package index and immutable KeepSafe Git source without probing internal pypicloud, then install the package editable without resolving dependencies again. Cache keys include requirements-dev.txt.
  • Aligns lxml==6.0.2 with libks==1.0.5, the source of truth for the email-service downstream environment.

Proof run

  • python3.11 --version: Python 3.11.13
  • Exact editable development install: pass
  • Clean CI=1 make ci-dev-install: pass using package-index dependencies and the immutable internal html-structure-diff commit, with no internal pypicloud probe
  • make lint: pass
  • make test: pass, 65 tests, 1 skipped, coverage total 84%
  • CI=1 make test: pass, 65 tests, 1 skipped; writes build/coverage/coverage.xml and build/test/results.xml
  • CI=1 make test-only: pass for the exact CircleCI test-job command
  • venv/bin/python -m compileall validator tests: pass
  • Exact-version import smoke for validator, validator.checks.url, aiohttp==3.13.2, beautifulsoup4==4.15.0, lxml==6.0.2, Markdown==3.10.2, parse==1.22.1, and commit-pinned sdiff==1.1.0: pass
  • venv/bin/content-validator --help and venv/bin/content-validator --version: pass
  • venv/bin/pip check: pass
  • Isolated sdist and wheel build plus metadata pin checks: pass; both artifacts preserve the immutable html-structure-diff commit reference
  • Downstream email-service editable integration on CPython 3.11.13: pass with libks==1.0.5, all four internal Python 3.11 worktrees, and lxml==6.0.2; pip check and pipdeptree --warn fail both pass
  • circleci config validate .circleci/config.yml: pass
  • circleci config process .circleci/config.yml: pass with CircleCI API access; reusable 2.1 config expands into the expected three jobs
  • Remote CircleCI on 365b74f: prepare_cache, lint, and test all passed, confirming the immutable internal sdiff reference resolves in the clean CI path
  • git diff --check: pass

Known gaps

  • Dependency install/build proof requires package-index and GitHub access; sdiff==1.1.0 currently resolves from an immutable internal commit and should move to an index pin after the KeepSafe release is published.
  • Email-service editable downstream validation is complete; package publication, generated requirements, and Ansible deployment updates remain follow-up work.
  • Service-only python311-service-upgrade-stack tasks remain intentionally skipped because this repo has no Paste/Gunicorn/INI/healthcheck/worker/Docker runtime surface.

Linear

BE-94: Upgrade content-validator to Python 3.11

Fixes BE-94

Modified Makefile test targets to use shared pynose coverage flags and CI XML/xunit artifact output.

Added migration contract details for the python311-service-upgrade-stack guardrail audit, no-stack deviations, dependency upgrade rationale, intentionally retained pins, and refreshed proof results.

Removed no files in this follow-up; it only documents and verifies the branch against the shared skill.
Bump remaining content-validator runtime pins to latest proven-safe versions: parse 1.22.0 and sdiff 1.0.0.

Adapt CircleCI from the python311-service-upgrade-stack sample with prepare_cache, lint, test, cache fallback keys, artifacts, and Codecov fallback.

Align Makefile CI install targets and refresh the migration contract with latest-version, msgpack not-applicable, CircleCI, and proof evidence.
Split restore_cache and save_cache option anchors so each CircleCI step receives only supported keys.
Pin aiohttp to the required 3.13.2 release and update beautifulsoup4, lxml, parse, and coverage to the approved current versions.

Keep requirements artifacts aligned and record the July compatibility proof in the migration contract.
Replace the legacy YAML-anchor layout with reusable CircleCI 2.1 executors and commands. Preserve the prepare, lint, test, cache, xUnit, coverage artifact, and non-fatal Codecov flow while using schema-correct restore and save cache steps. Record CLI validation and processed-config proof in the migration contract.
Drop the completed migration tool from the maintained development dependency set and requirements mirror. Remove its repo-local migration-contract references now that the Python 3.11 source modernization has been applied and verified.
Install committed development requirements from public sources before the editable package install, avoiding unreachable internal pypicloud probes in CircleCI. Add the shared KeepSafe SSH key fingerprint, refresh caches when development requirements change, and document project-side key provisioning.
Replace the unavailable public-PyPI sdiff==1.1.0 pin with an immutable reference to the reviewed html-structure-diff Python 3.11 commit. Preserve the lxml 6.0.2 downstream constraint and record clean CI-install, test, wheel-install, metadata, and dependency-integrity proof.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant