Skip to content

Fix ICM 51000001082652: VMBackup extension fails to import on Python 3.13+ - #2196

Open
vupadhyay-ms wants to merge 3 commits into
Azure:masterfrom
vupadhyay-ms:dev/vupadhyay/linux-ext-bug-fix
Open

Fix ICM 51000001082652: VMBackup extension fails to import on Python 3.13+#2196
vupadhyay-ms wants to merge 3 commits into
Azure:masterfrom
vupadhyay-ms:dev/vupadhyay/linux-ext-bug-fix

Conversation

@vupadhyay-ms

@vupadhyay-ms vupadhyay-ms commented Jul 10, 2026

Copy link
Copy Markdown

The Issue

Customer VM (Ubuntu 26.04 / Py 3.14) fails backup. Extension log shows:

File ".../WaagentLib.py", line 26, in <module>
    import crypt
ModuleNotFoundError: No module named 'crypt'

File ".../WAAgentUtil.py", line 69, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

File "main/handle.py", line 40, in <module>
    from mounts import Mounts

The 3 stdlibs

Module Deprecated REMOVED in Used in Fails on
crypt Py 3.11 Py 3.13 WaagentLib.py:26 (bare import crypt) Ubuntu 26.04 / Py 3.14
distutils Py 3.10 Py 3.12 WaagentLib.py:59 (from distutils.version import LooseVersion) Debian 13, RHEL 10, Fedora 39+
imp Py 3.4 Py 3.12 WAAgentUtil.py:69 (import imp fallback) Same as above

Python removed these modules. Our code imports them unconditionally → extension crashes at file-load time → VMExtensionProvisioningTimeout → customer backup fails.


History (why previous fix was reverted)

  1. Nov 2025PR #2124 tried the same fix. Merged.
  2. Apr 2026 — Broke Py 2.7 VMs → ICM 783505554 (Sev2). Root cause: 2 subtle bugs:
    • Unicode in fallback class docstring → Py 2 SyntaxError at parse time
    • except ImportError failed to catch AttributeError from importlib.util.module_from_spec on Py 2.7
  3. Apr 2026PR #2163 fully reverted Updates for deprecated modules in Python 3.12 and Invalid escape sequence issues #2124.
  4. Result — original Py 3.13+ crash returned → this ICM.

This PR re-lands the fix with those 2 bugs corrected.


The Fix — 5 edits, 2 files

VMBackup/main/WaagentLib.py

Edit 1 — Add PEP 263 encoding declaration

# -*- coding: utf-8 -*-

Prevents future Unicode-in-source regressions (prevents ICM 783505554 class of bug).

Edit 2 — Wrap import crypt

try:
    import crypt
except ImportError:
    crypt = None

crypt is dead code (VMBackup never calls gen_password_hash). Wrapping stops the crash.

Edit 3 — Replace distutils.LooseVersion with 3-tier ladder

try:
    from packaging.version import Version as LooseVersion       # Tier 1: modern
except ImportError:
    try:
        from distutils.version import LooseVersion              # Tier 2: legacy stdlib
    except ImportError:
        class LooseVersion(object): ...                          # Tier 3: ASCII-only fallback

Guarantees LooseVersion resolves on every Python 2.7 → 3.14. Tier 3 class is identical to PR #2124's, with 2 corrections: (a) inherits from object for Py 2.7 rich-comparison correctness, (b) ASCII -> instead of Unicode .

Edit 4 — Defensive guard in gen_password_hash()

if crypt is None:
    raise NotImplementedError("gen_password_hash requires the 'crypt' stdlib module...")

Never hit today; fails loudly if some future caller reaches this on Py 3.13+.

VMBackup/main/Utils/WAAgentUtil.py

Edit 5 — Replace exception-based fallback with capability check

if importlib_util is not None and hasattr(importlib_util, 'module_from_spec'):
    # Modern path — Py 3.5+
    spec = importlib_util.spec_from_file_location('waagent', agentPath)
    waagent = importlib_util.module_from_spec(spec)
    spec.loader.exec_module(waagent)
else:
    # Legacy path — Py 2.6–3.4 only
    import imp
    waagent = imp.load_source('waagent', agentPath)

Testing —

Test Setup: Azure Ubuntu 24.04 VM + python3.10 / 3.12 / 3.13 / 3.14 (via deadsnakes PPA) + Docker python:2.7 for regression coverage.

Test Py 2.7 Py 3.10 Py 3.12 Py 3.13 Py 3.14
py_compile on both files
Full import chain
handle.py -enable exit 0
image image

Vinayak Upadhyay and others added 2 commits July 10, 2026 16:28
… 51000001082652)

Ubuntu 26.04 / Py 3.14 removed the stdlib modules 'crypt' (Py 3.13),
'distutils' (Py 3.12), and 'imp' (Py 3.12). VMBackup imports all three
unconditionally, so handle.py -enable crashes at file-load time before
any snapshot logic runs -> CRP reports VMExtensionProvisioningTimeout ->
customer backup fails.

Changes (2 files):

* VMBackup/main/WaagentLib.py
  - Add PEP 263 encoding declaration (defense-in-depth vs future Unicode)
  - Wrap 'import crypt' in try/except; crypt = None fallback
  - Replace 'from distutils.version import LooseVersion' with 3-tier ladder:
    packaging.version -> distutils.version -> ASCII-only hand-written class
  - Guard gen_password_hash() with 'if crypt is None: raise NotImplementedError'

* VMBackup/main/Utils/WAAgentUtil.py
  - Replace 'except ImportError' fallback with hasattr(importlib.util,
    'module_from_spec') capability check (Py 2.7 raised AttributeError,
    not ImportError - this was one of the PR Azure#2124 regressions)
  - Drop outer 'except Exception: raise Exception(Cant load waagent)'
    wrapper that was hiding the real traceback in ICM logs

This is a corrected re-landing of PR Azure#2124 (reverted by PR Azure#2163 due to
ICM 783505554). Defenses vs those regressions:
  - Pure ASCII in modified files (byte-scan verified: 0 non-ASCII bytes)
  - PEP 263 encoding declaration future-proofs against Unicode leaks
  - hasattr() capability check is exception-type-independent
  - Tier 3 LooseVersion class inherits from 'object' (new-style, correct
    Py 2.7 rich-comparison dispatch)

Tested on Py 2.7 (Docker), 3.10, 3.12, 3.13, 3.14: 18/18 PASS.
End-to-end portal-triggered backup on Py 3.14 with branch code in
/var/lib/waagent/... completed with extension status: success and
snapshot URIs written to blob storage.

Fixes: ICM 51000001082652
Re-lands: PR Azure#2124 (reverted in PR Azure#2163)
Regression-tests: ICM 783505554
Related: Azure#2172

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Encoding declaration retained; only the 4-line comment block above it removed per review feedback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@vupadhyay-ms
vupadhyay-ms requested a review from a team as a code owner July 10, 2026 11:39
@vupadhyay-ms

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

Combine the strengths of both fallback shims discussed in review:

- Keep pre-release precedence (alpha < beta < rc < release) via negative
  sentinels and the [.\-_] separator split (broader than a digits-only
  tokenizer for kernel-style strings).
- Add explicit str() coercion when tuple positions have mismatched types
  so Py 3 does not raise TypeError comparing int vs str (e.g. '1.0.5' vs
  '1.0.foo'). This matches the safe pattern from PR Azure#2177.
- Add __ne__ so Py 2.7 does not fall back to identity comparison; Py 3
  auto-derives it but Py 2.7 does not.
- Add __repr__ for cleaner logs and debugging.
- Fold six near-identical rich-comparison methods into a single _cmp()
  helper and one-line delegates, halving the surface area.

No behavior change for the 4 in-tree call sites (all pass numeric plugin
version strings such as '1.0.9231.0'); shim now degrades safely on
arbitrary inputs instead of raising.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5519adfb-3554-4ee6-84ff-61ec7bab7e1a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant