Yahoo Ads: Migrate to OpenRTB 2.6#4521
Conversation
Bump the x-openrtb-version header from 2.5 to 2.6 and stop down-converting the auction request inside the bidder. The bidder now passes the 2.6 request through, relying on the PBS-Java up-converter to normalize gdpr, us_privacy, consent, eids, schain and rwdd to their 2.6 top-level slots. For the three privacy/regulatory fields that neither converter handles (gpp, gpp_sid, coppa), add ext->top promotion in modifyRegs: read the 2.6 top-level value, falling back to the legacy 2.5 regs.ext property, and strip the promoted keys from ext. Typed ext-only fields (gpc, dsa) and any other ext properties are preserved. No cattax default is synthesized. Remove the now-unused BidRequestOrtbVersionConversionManager dependency from the bidder and its Spring configuration. Update unit tests for the 2.6 wire shape and add coverage for the ext->top promotion, mixed publisher shapes, wrong-type ext guards, and the no-ext short-circuit. Update the integration fixture to the 2.6 wire shape.
Make stripPromotedFromExt remove a regs.ext key only when its value was actually promoted to top-level (resolved value non-null). A malformed, non-promotable value (e.g. a non-textual gpp) is now left untouched in regs.ext instead of being dropped when a sibling field triggers the rebuild. Add a test covering the mixed case (valid coppa promoted, malformed gpp kept in ext) and trim redundant comments.
resolveGppSid now promotes regs.ext.gpp_sid only when every element is an integer. If any element is non-integral the whole array is left in regs.ext untouched rather than promoting a filtered subset, matching the gpp and coppa handling so no value is silently dropped. Adds a test for the mixed-array case.
Restructure the regs handling so a single per-field signal decides both the promotion and the ext strip. A field is promoted only when it is absent at top-level and present and well-formed in ext; a key is removed from regs.ext if and only if that field was promoted. This removes the previous dependency on whether a sibling field triggered the rebuild (e.g. a value present at both top-level and in ext is now handled the same regardless of siblings). Rename for clarity: modifyRegs -> promoteRegsExtToTopLevel, resolveX -> xToPromote (return the value to lift or null), stripPromotedFromExt -> removePromotedKeys. Add a test for the same field present at both top-level and ext.
|
Hi @osulzhenko , @Fuska1 , @bukrain and @gmiedlar-ox any update on this PR? Thank you. |
Hello. The team is actively working on preparing the 4.0 release. As this requires extensive testing, the codebase is temporarily frozen to not create additional issues that will further delay the 4.0 release. Be assured that the review for this PR is in the queue. Thanks for understanding. |
Thanks @Net-burst! Any rough timeline for the 4.0 release? |
There was a problem hiding this comment.
I'm not sure you need all the promote* methods, since we already do that for you (see BidRequestOrtb25To26Converter).
Also, is it even possible for gpp, gppSid, and coppa to be in regs.ext? We don't even read them from there in the privacy services.
There was a problem hiding this comment.
Thanks for the review @CTMBNara.
I looked into BidRequestOrtb25To26Converter on this. It promotes gdpr and us_privacy up from regs.ext, but it doesn't touch gpp, gpp_sid, or coppa, so those three don't get promoted for us anywhere in core. Let me know if I've missed something there.
Main reason I kept the promotion was it's a no-op for normal requests (values are already top-level, so it skips), but if any arrive under regs.ext they'd otherwise stay nested and get dropped on the way out. Since these are GPP/COPPA signals, I'd rather not risk silently losing one during the cutover. It only kicks in when the top-level field is empty, and leaves everything else in regs.ext untouched.
Resolve conflict in YahooAdsConfiguration by combining upstream's removal of the yahooAds usersyncer with this branch's removal of the OpenRTB version conversion manager, leaving the two-arg YahooAdsBidder(endpoint, mapper).
scr-oath
left a comment
scr-oath
left a comment
There was a problem hiding this comment.
OpenRTB 2.6 migration — review
Headline first: the core design here is correct and necessary, and I want to explicitly back you up on the earlier thread. I independently traced BidRequestOrtb25To26Converter — it promotes only gdpr/us_privacy out of regs.ext and leaves gpp, gpp_sid, and coppa buried as flexible ext properties. So the promote* logic is not redundant with core. The helpers are pure/immutable and well-isolated, and the unit suite is genuinely thorough. 👏
Requesting changes on a small, targeted set:
Should fix before merge
- Silent int-narrowing of
gpp_sid/coppa(see inline).isIntegralNumber()acceptslong/BigInteger, andasInt()then truncates silently — an oversized element passes the "malformed?" guard and ships as a wrong (possibly negative) section ID instead of being left alone.canConvertToInt()(already used inRubiconBidder) closes it in one line. Low probability, but it's privacy data and it defeats the leave-it-untouched guarantee the method advertises. - The integration test doesn't exercise the new promotion path (see inline on the fixture). The incoming IT request already carries
gpp/gpp_sidtop-level, so only core'sgdprpromotion runs and the bidder short-circuits onext == null. Unit coverage is excellent, but the IT gives false confidence for this PR's stated purpose. Concrete fixture edit suggested inline.
Worth a decision (not hard blockers)
3. Stale ext copy left on the wire when the top-level field is already set (see inline) — can put two differing values for one consent signal on the request. It's deliberate + tested, but diverges from core's stricter "once superseded, always clear" (resolveRegsExt). I'd match that pattern.
4. Reachability (the crux of @CTMBNara's second question): AuctionGppService reads gpp/gpp_sid top-level only (no regs.ext fallback), so PBS's own privacy enforcement never sees an ext-nested GPP string. Could you note the concrete upstream request shape that actually delivers gpp/gpp_sid/coppa under regs.ext after core conversion? That would nail down whether the promotion case is reachable in practice.
Follow-ups (separate issues, not this PR)
- Consider promoting
gpp/gpp_sid/coppaonce inBidRequestOrtb25To26Converterso everyortb-version: "2.6"bidder benefits, rather than per-bidder. AuctionGppServicelacking aregs.extfallback for GPP looks like a core-level gap worth its own issue.
Also: the PR currently shows as CONFLICTING — it'll need a rebase on master.
External confirmation (likely yours/Yahoo's to own): dropping the full down-conversion now also sends user.consent/user.eids/source.schain/imp.rwdd top-level. The e2e HTTP 200s are reassuring, but a 200 doesn't prove the SSP honors the TCF string at its new location — worth a signal-level sign-off given the yaml already declared 2.6.
Thanks for the excellent write-up — the field-mapping table and wire-shape matrix made this much easier to review.
| } | ||
| final List<Integer> sids = new ArrayList<>(node.size()); | ||
| for (final JsonNode elem : node) { | ||
| if (!elem.isIntegralNumber()) { |
There was a problem hiding this comment.
issue (non-blocking): isIntegralNumber() is true for long/BigInteger nodes, and asInt() on those silently truncates (3000000000 → -1294967296, 2147483648 → -2147483648). So an out-of-int-range gpp_sid element passes this "malformed?" guard and then gets shipped to the SSP as a corrupted section ID — which defeats the "leave it in ext untouched" guarantee this method is built around. Same narrowing applies to coppa at line 230.
Suggest elem.canConvertToInt() / node.canConvertToInt() (already the idiom in RubiconBidder) — it returns false for both non-integral and out-of-range values, so oversized input falls through to the safe "leave untouched" path instead of emitting wrong data.
| } | ||
|
|
||
| // Rebuild regs.ext keeping the typed fields and every property except the promoted ones. | ||
| private static ExtRegs removePromotedKeys(ExtRegs ext, |
There was a problem hiding this comment.
suggestion (design): when a field is already set top-level and a different value also sits in regs.ext, the ext copy is left in place — so the outbound request can carry two conflicting declarations of the same consent signal. This is deliberate and well-tested (makeHttpRequestsShouldKeepExtGppWhenTopLevelGppAlreadySetEvenIfSiblingIsPromoted), but it diverges from core's stricter rule in BidRequestOrtb25To26Converter.resolveRegsExt, which clears the ext copy of gdpr/us_privacy whenever ext had a value at all ("once superseded, always clear").
Consider matching that here: strip the ext key whenever a valid top-level value exists, not only when this call performed the promotion. Then the wire can never carry two values for one signal.
| private static MultiMap makeHeaders(Device device) { | ||
| final MultiMap headers = HttpUtil.headers() | ||
| .add(HttpUtil.X_OPENRTB_VERSION_HEADER, "2.5"); | ||
| .add(HttpUtil.X_OPENRTB_VERSION_HEADER, "2.6"); |
There was a problem hiding this comment.
suggestion (non-blocking): this hard-coded "2.6" duplicates ortb-version: "2.6" in yahooAds.yaml with no link between them — which is exactly the drift that produced the bug this PR fixes (yaml said 2.6, the code did 2.5). It matches the codebase's dominant convention, so not a new problem, but at minimum consider hoisting it to a named constant next to GPP_PROPERTY (newer bidders like Msft do this). A nice follow-up would be setting this header centrally from BidderInfo.getOrtbVersion() so the ~38 hand-maintained copies across bidders collapse to one source of truth.
|
|
||
| if (regs != null) { | ||
| requestBuilder.regs(modifyRegs(regs)); | ||
| requestBuilder.regs(promoteRegsExtToTopLevel(regs)); |
There was a problem hiding this comment.
nitpick (non-blocking): promoteRegsExtToTopLevel(regs) is a pure function of regs, which is captured once before the imp loop and never changes across iterations — but it's recomputed (including a full ext.getProperties() walk) once per imp. For multi-imp requests, consider hoisting it out of the loop in makeHttpRequests and passing the already-promoted Regs into modifyRequest.
| "gpp_sid": [6], | ||
| "gdpr": 0 | ||
| } | ||
| "gpp": "gppstring", |
There was a problem hiding this comment.
suggestion (non-blocking): heads-up that this fixture change is bookkeeping for the removed down-conversion, not coverage of the new promotion logic. The incoming fixture test-auction-yahooads-request.json already has gpp/gpp_sid top-level, and its only ext field (gdpr) is promoted by core (BidRequestOrtb25To26Converter) before the bidder runs — so promoteRegsExtToTopLevel short-circuits on ext == null and the new gpp/gpp_sid/coppa path is never hit end-to-end.
To actually exercise it, nest them in the incoming request:
"regs": { "ext": { "gdpr": 0, "gpp": "gppstring", "gpp_sid": [6], "coppa": 0 } }and expect the flattened top-level shape here (using coppa: 0 doubles as a falsy-zero regression guard).
4 participants
🔧 Type of changes
Migrates the Yahoo Ads bidder (and its
yssp/yahoossp/yahooAdvertisingaliases) from OpenRTB 2.5 to OpenRTB 2.6 on the wire. Three changes, in the
bidder and its Spring config:
x-openrtb-version: 2.5→2.6insrc/main/java/org/prebid/server/bidder/yahooads/YahooAdsBidder.java.request back to 2.5 before serializing it
(
conversionManager.convertFromAuctionSupportedVersion(bidRequest, OrtbVersion.ORTB_2_5)).That call is removed, along with the now-unused
BidRequestOrtbVersionConversionManagerdependency in the bidder and itsYahooAdsConfiguration.regs.{gpp, gpp_sid, coppa}. The oldmodifyRegs/resolveExtRegsblock re-buried these (and gdpr/us_privacy)into
regs.ext. It is replaced withpromoteRegsExtToTopLevel: a field ispromoted only when it is absent at top-level and present and well-formed in
regs.ext, and a key is removed fromregs.extif and only if that fieldwas promoted. Typed ext-only fields (
gpc,dsa), unknownregs.extproperties, a value already present at top-level, and any of the three that
could not be promoted (e.g. a malformed, non-textual
gpp, or agpp_sidarray with a non-integer element) are all left in
regs.extuntouched, sono value is silently dropped. The strip decision uses a single per-field
signal, so it does not depend on whether a sibling field triggered the
rebuild.
No publisher-facing config change is required bidder params (
dcn,pos) areunchanged, and
bidder-config/yahooAds.yamlalready declaredortb-version: "2.6".Why
PBS-Java core up-converts every incoming request to its canonical 2.6 form
(
BidRequestOrtb25To26Converter) before bidders run. The Yahoo bidder thenexplicitly down-converted that back to 2.5 and re-buried
gpp/gpp_sid/coppaintoregs.ext, and sentx-openrtb-version: 2.5.Net effect: even fully 2.6-aware publishers had privacy/addressability fields
buried in
.exton the wire.Removing the in-bidder down-conversion lets core's existing promotion reach the
wire. The new
promoteRegsExtToTopLeveladditionally promotes the threeregulatory fields that neither converter handles.
regs.gdpr,regs.us_privacyregs.ext.*Regsuser.consent,user.eidsuser.ext.*Usersource.schainsource.ext.schainSourceimp.rwddimp.ext.prebid.is_rewarded_inventoryImpregs.gpp,regs.gpp_sidregs.ext.*Regsregs.copparegs.ext.coppa(legacy)RegsWhat this means for publishers
regs.gpptop-level).ext; header was2.52.6.exton the wiregdpr,us_privacy,consent,eids,schain,rwdd; the bidder additionally promotesgpp,gpp_sid,coppa; everything else passes through in.extNon-regulatory 2.6 fields the bidder does not transform (e.g.
device.sua,user.kwarray,site/app.inventorypartnerdomain,content.network,content.channel) pass through in whatever shape the publisher sent.What changed
src/main/java/org/prebid/server/bidder/yahooads/YahooAdsBidder.javasrc/main/java/org/prebid/server/spring/config/bidder/YahooAdsConfiguration.javasrc/test/java/org/prebid/server/bidder/yahooads/YahooAdsBidderTest.javasrc/test/resources/org/prebid/server/it/openrtb2/yahooads/test-yahooads-bid-request.jsonYahooAdsBidder.javabumps the header literal, removes the down-converter calland dependency, and replaces the regs re-bury logic with
promoteRegsExtToTopLevel+gppToPromote/gppSidToPromote/coppaToPromote(each returns the value to lift, or null) andremovePromotedKeys. The helpers operate on atoBuilder()copy ofRegs(the bidder already rebuilds the request per-impression), so they cannot affect
the request other bidders see.
Test plan
Automated
mvn test -Dtest=YahooAdsBidderTest29 unit tests pass. New cases cover:ext→top promotion, mixed publisher shapes (some fields top-level, some in
ext), wrong-typed
regs.extvalues (no promotion), a malformed value left inext while a sibling field promotes, a
gpp_sidarray with a non-integerelement left in ext untouched, the same field present at both top-level and
ext (ext copy left untouched), and the no-
extshort-circuit.mvn test -Dtest=YahooAdsTestintegration test passes against theupdated 2.6 wire fixture.
End-to-end against the Yahoo Ads SSP
A build of this branch was run as a local PBS-Java process (not Docker) and
pointed at the Yahoo Ads SSP endpoint. For each scenario a request was POSTed to
/openrtb2/auctionwithext.prebid.debug: 1, and the exact bytes the bidderput on the wire were read back from
ext.debug.httpcalls.yahooAds[*](requestbody + headers). Every outbound carried
x-openrtb-version: 2.6and the SSPreturned HTTP 200. Bid presence varies with demand and is not asserted here.
regs.gpp/regs.gpp_sid/regs.coppapromoted to top-level and removed fromregs.ext; core-promoted fields (gdpr,us_privacy) at top-level; unrelatedregs.extkeys (gpc,dsa) left inregs.extplcmt=2andplcmt=4podid,podseq,slotinpod,maxseq,poddur,mincpmpersec,rqddurs) intactapp.idoverride applied (same mechanism assite.id); 2.6 wireHow to verify locally
Test fixture changes
test-yahooads-bid-request.json(integration): expected outbound now carriesregs.gpp/regs.gpp_sid/regs.gdprat top-level (previously underregs.ext).YahooAdsBidderTest.java: header assertion2.5→2.6; the"remove/overwrite regs ext" tests are replaced with promotion + passthrough
tests described above.
Backward compatibility
No breaking changes.
dcn,pos) unchanged.makeBids) unchanged.gpp/gpp_sid/coppaat top-level; all other fields pass through unchanged in.ext.Aliases
yssp,yahoosspandyahooAdvertisingare declared as aliases ofyahooAdsin
bidder-config/yahooAds.yamland share the sameYahooAdsBidderinstance,so all alias codes send the 2.6 wire after this PR.
For contact: prebid-tech-team@yahooinc.com
Implemented with assistance from Claude Code.