fix: skip RSA_MIN_PAD_SZ check for PSS padding in RsaPublicEncryptEx

[MarkAtwood]

Summary

The RSA_MIN_PAD_SZ guard (inLen > sz - 11 → RSA_BUFFER_E) is a PKCS#1 v1.5 concept. PSS has its own length check inside RsaPad_PSS (emLen >= hLen + sLen + 2 per RFC 8017 §9.1.1) and the outer guard fires first, before RsaPad_PSS ever runs.

For keys in the range [hLen+2, hLen+10] bytes, the outer guard incorrectly returns RSA_BUFFER_E for combinations where PSS with saltLen=0 would be geometrically valid. Keys in this range are non-standard but valid — they can be loaded from external DER.

Fix: add a WC_RSA_PSS #ifdef that skips the RSA_BUFFER_E return when pad_type == WC_RSA_PSS_PAD, mirroring the existing WC_RSA_NO_PADDING exception for raw mode.

Test plan

• PSS encrypt with non-standard key sizes in [hLen+2, hLen+10] bytes no longer returns RSA_BUFFER_E
• PSS still correctly rejects combinations that exceed its own length constraint (PSS_SALTLEN_E)
• Existing RSA PSS tests unaffected

/cc @wolfSSL-Fenrir-bot please review

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adjusts RsaPublicEncryptEx so the RSA_MIN_PAD_SZ (PKCS#1 v1.5) input-length guard does not preempt PSS padding’s own size validation, enabling certain non-standard small key sizes to proceed to RsaPad_PSS.

Changes:

• Skips the RSA_BUFFER_E early-return when pad_type == WC_RSA_PSS_PAD (under WC_RSA_PSS).
• Keeps the existing exception path for WC_RSA_NO_PAD behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[MarkAtwood]

Restructured as a single compound condition — pushed f05d929. No more implicit nesting of preprocessor-guarded if-statements. Each exemption is now a && clause gated by its own #ifdef, so adding future exemptions is just another line.

[MarkAtwood]

Restructured per review feedback — the implicit preprocessor nesting is replaced with a check_min_pad flag pattern (91a1b28). Each exemption is a separate #ifdef block, and there's a single guarded return RSA_BUFFER_E. Ready for re-review.

[dgarske]

Skoll Code Review

Scan type: reviewOverall recommendation: COMMENT
Findings: 2 total — 2 posted, 0 skipped
2 finding(s) posted as inline comments (see file-level comments below)

Posted findings

• [Medium] New brace-scoped block introduced solely to declare check_min_pad — wolfcrypt/src/rsa.c:3700-3715
• [Medium] New PSS small-key code path has no added test coverage — wolfcrypt/src/rsa.c:3707-3711

Review generated by Skoll

[dgarske]

CI is unhappy

wolfcrypt/test/test.c: In function ‘rsa_pss_test’:
wolfcrypt/test/test.c:27726:22: error: stack usage might be 33328 bytes [-Werror=stack-usage=]
27726 | static wc_test_ret_t rsa_pss_test(WC_RNG* rng, RsaKey* key)
      |                      ^~~~~~~~~~~~
wolfcrypt/test/test.c:28155:1: error: the frame size of 33248 bytes is larger than 4096 bytes [-Werror=frame-larger-than=]
28155 | }
      | ^
cc1: all warnings being treated as errors

[MarkAtwood]

@dgarske Fixed — rsa_pss_test was putting a full RsaKey on the stack, which blows up to ~33 KB under --enable-smallstack --with-max-rsa-bits=16384. Moved it to the heap using the same WOLFSSL_SMALL_STACK pattern as dh_test; frame drops from 33216 to 368 bytes (-fstack-usage), assertions unchanged. Also rebased onto current master (branch was ~3 months behind). Small-stack build: testwolfcrypt passes; non-small-stack branch compiles clean.