Improve numerical robustness of hinfnorm/linfnorm

[baggepinnen]

Background

hinfnorm intermittently failed in CI, most recently in run 29119351321 at test_conversion.jl:221, where the norm of an analytically-zero system difference evaluated to Inf.

Root-causing this revealed two separate things:

1. The specific CI failure is not a hinfnorm bug. minreal(::StateSpace) with its default tolerance (atol=0, rtol-based rank tests) occasionally leaves a weakly observable mode at/just outside the unit circle. Non-normality amplifies the modal residue of the leftover mode (the eigenvector condition number is ~1e4 for the failing realization), and evaluating the frequency response confirms the realization genuinely has gain ≈ 15 near the boundary — Inf is the correct H∞ answer for the system hinfnorm was handed. Whether the mode survives minreal depends on machine/BLAS rounding, hence the intermittency. The test now reduces with an explicit minreal tolerance, which deterministically removes the junk modes (verified over 100 perturbed reruns), and asserts against a threshold above that tolerance since truncated modes may contribute gain of the order of the truncation tolerance.

2. The two-step algorithm had several genuine robustness weaknesses in adjacent scenarios, addressed here.

Changes to hinfnorm/linfnorm

• Residue-aware boundary/instability decisions. Poles on/outside the stability boundary whose modal residue is below resid_tol*opnorm(B)*opnorm(C) are treated as spurious remainders of imperfect pole-zero cancellations and are ignored instead of forcing Inf (e.g., the norm of s/s now evaluates to 1). New keyword resid_tol, default nx*√eps; resid_tol = 0 recovers the strict behavior. Residues are estimated from the eigenvector decomposition of the (Schur-form) A; if the decomposition fails, all suspect poles are conservatively treated as genuine.
• Crossing-point evaluation + stagnation termination in the γ-iteration. The gain is evaluated at the detected crossing frequencies in addition to midpoints, guaranteeing progress when only a single crossing is detected (previously a silent infinite loop until maxIters in continuous time, and a thrown error in discrete time). If an iteration fails to improve the lower bound beyond the γ-margin, the iteration terminates: this handles tangential crossings (converged) and crossing detections that are artifacts of spurious near-boundary poles.
• Relative imaginary-axis tolerance. abs(real(z)) < approximag*max(1, abs(z)) for both the pole check and Hamiltonian-eigenvalue crossing detection, so the test remains meaningful for fast dynamics (crossings at large |λ| were previously missed because eigenvalue rounding scales with |λ|). This also resolves the old in-code QUESTION comment about slow dynamics.
• Non-finite guards on all gain evaluations (evalfr at a frequency essentially coinciding with a pole returns non-finite values that previously could poison the lower bound).
• Better initial lower bound: the gain is evaluated at the frequency and magnitude of every pole (cheap, since A is triangular after the Schur reduction). This reduces iteration counts and is required so that systems whose gain is singular at the previous candidate frequencies do not start from a zero lower bound.
• Missing zero-gain early return added to the discrete-time path (previously a zero system reaching the iteration would build a singular R).
• Unified non-convergence handling: both time domains now @warn and return the best estimate; the discrete-time path previously threw a hard error, and the continuous-time message misidentified itself as _dt.
• Cleanup of the discrete-time initialization (stale comment, unused variable).

Tests

• 13 new regression cases in test_linalg.jl: spurious vs. genuinely coupled boundary/unstable modes (both time domains, including small-absolute-gain systems where the coupling is genuine), resid_tol=0 strictness, and termination for exactly-decoupled boundary modes.
• test_conversion.jl: the flaky assertions now use minreal(..., 1e-9) with thresholds above the truncation tolerance (see background).
• test_matrix_comps.jl: the gain peak of S_test is flat, so the peak frequency is only determined to within ~w*√tol (the previous reference value was itself 8e-4 from the true argmax); the frequency tolerance at default tol is relaxed to 1e-2 and a tight assertion at tol=1e-10 is added.

Verification

• Full ControlSystemsBase suite passes locally (21181 pass, 6 pre-existing broken), including test_implicit_diff (differentiation through hinfnorm unaffected; the extension forwards kwargs...).
• The previously flaky scenario passes 100/100 under parameter perturbations far larger than machine-level rounding variation.
• The failing CI realization was reproduced locally and used to validate the analysis and the fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_014nRhRodrFbP41QBEDQ9hTw

[JuliaControlBot]

This is an automated message.
Plots were compared to references. 4/11 images have changed, see differences below.
After pulling this PR, please update the reference images by creating a PR to ControlExamplePlots.jl here.

Difference	Reference Image	New Image
✔️ 0.0
✔️ 0.0
✔️ 0.0
✔️ 0.0

[codecov]

Codecov Report

❌ Patch coverage is 93.10345% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.67%. Comparing base (b2b7dc4) to head (e016b11).

Files with missing lines	Patch %	Lines
lib/ControlSystemsBase/src/matrix_comps.jl	93.10%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1066      +/-   ##
==========================================
+ Coverage   91.64%   91.67%   +0.03%     
==========================================
  Files          42       42              
  Lines        5660     5707      +47     
==========================================
+ Hits         5187     5232      +45     
- Misses        473      475       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.