fix(train): send bounded HyperParameters override map for serverless customization jobs

[jam-jee]

Issue

Refs P467902218.

Serverless (SMTJ) model-customization jobs (RLVR/SFT/DPO/MTRL) began failing at CreateTrainingJob after the Nova release with:

ValidationException: 1 validation error detected: Value '{freeze_vision_model=True, ...,
learning_rate=1e-06}' at 'hyperParameters' failed to satisfy constraint:
Member must have length less than or equal to 100

A notebook that previously worked started failing (~07/06). The SDK was sending all hyperparameters — including recipe defaults the user never changed.

Root cause

The Nova release added recipe/overrides support and a new _apply_recipe_to_hyperparameters() step on the serverless training path. For serverless jobs it flattened the entire resolved Hub recipe (200+ leaf keys — KL config, vLLM params, LoRA settings, etc.) into the hyper_parameters map passed to CreateTrainingJob.

The serverless HyperParameters map is an override map capped at 100 entries by the API (the error message is a Java Map.toString() echo of the whole map, and the "length ≤ 100" constraint is the entry-count limit). Flattening the full recipe blew past that limit. Pre-Nova, serverless only sent the bounded spec subset (to_dict()), which is why the same notebook worked before.

Fix

For the serverless path (new serverless=True flag on _apply_recipe_to_hyperparameters), emit a bounded override map containing only:

• Hub spec-allowlisted keys, plus
• leaves the user explicitly changed (via overrides=, a recipe file, or direct .hyperparameters.* assignments).

The recipe's unchanged non-spec defaults are dropped, since the base recipe is applied server-side. Crucially, explicit user overrides of non-spec keys are still forwarded — they are never silently dropped (which would train with the wrong value). This mirrors the delta approach the MTRL trainer already uses.

The serverful path is unchanged (serverless=False default): it keeps full-recipe flattening because those values are rendered back into the recipe YAML handed to ModelTrainer.from_recipe.

Applied to RLVRTrainer, SFTTrainer, DPOTrainer, and MultiTurnRLTrainer.

Testing

• Added 3 regression tests in test_serverful_recipe_validation.py covering: unchanged non-spec defaults are dropped for serverless, explicit user overrides (incl. deeply nested) are forwarded, and the serverful path still flattens the full recipe.
• Updated 3 tests in test_trainer_recipe_integration.py that had locked in the pre-fix behavior (full recipe flowing into serverless hyper_parameters) to assert the corrected override-map contract.
• Full tests/unit/train suite: 2152 passed, 19 skipped. (One unrelated pre-existing failure, test_model_trainer_no_from_recipe_raises, fails identically on clean master — it needs AWS creds for a default-bucket lookup.)

Merge Checklist

• I have read the CONTRIBUTING doc
• I used the commit message format described in CONTRIBUTING
• I have passed the region in to all S3 and STS clients that I've initialized as part of this change (N/A)
• I have updated any necessary documentation, including READMEs and API docs (N/A)

Tests

• I have added tests that prove my fix is effective or that my feature works
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
• I have checked that my tests are not configured for a specific region or account (N/A)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.