feat(openai): send images to the GPT Live backend model - #7316
Open
tinalenguyen wants to merge 4 commits into
Open
tinalenguyen wants to merge 4 commits into
tinalenguyen wants to merge 4 commits into
Conversation
The voice model has no image channel, so an image goes to the backend Responses model as an input item. Two ways in, both ordinary ones: - push_video implements the DuplexSession hook, so RoomInputOptions(video_enabled=True) now reaches GPT Live rather than hitting the base class no-op. - an llm.ImageContent in the chat context becomes a Responses image input item, with the message's own words as its caption. A url the backend can fetch itself is passed through, not re-encoded. Nothing runs on its own: the image waits in the backend's input until the voice model next delegates, or a tool result continues it. Under delegation="client" there is nothing to look at one, so video warns once a session and a context image warns once a message, while its words still reach the voice model. Queued events now carry the connection they were written for. Audio appends, response.item.create and response.create are dropped rather than replayed into a later session, whose backend has none of the state they name; the backend kinds warn so the caller knows to send them again. Context appends still ride over, since the voice model is just as happy to hear them on the next connection. Refs #7291
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… on its session Two holes in the image and reconnect handling. An image already in the chat context reached nobody. session.start carries the conversation so far, but its input items are text only, and _append_items returns before routing anything while the session has yet to start. So an agent seeded with "What is shown?" and a screenshot sent the words and lost the picture. The images are now queued behind the event that carries their words, composed per connection like the rest of the startup history, so the session replacing a dropped one gets them too. A queued context append scoped to a delegation could cross connections. A delegation belongs to the session that created it, so replaying its answer into the replacement cannot continue the work it names. _keep_stale_event now treats any append with a delegation_id as connection bound; an unscoped append is still context the next session is happy to hear. The startup path collects its images in the loop that renders the history rather than in a second pass, and the one warning for a session with no backend to look at an image now lives in _send_backend_input. Refs #7291
…keep their role Three things the image path got wrong, and one of them was mine twice over. A message carrying only an image was skipped whole. _render_item returns None when there are no words, and the loop moved on before collecting the picture, so the fix for startup images covered the captioned case and left the bare one exactly as broken as before. The images are now taken before that decision. An image kept only when the caller sent it. The reason given for that was wrong: a Responses input item takes user, system and developer, so a standing reference image on a system message has a perfectly good way in and was being dropped for no reason. It now travels under the role that carried it. Only assistant is left out, since an assistant turn is output and the API has no input item for one. The item itself is no longer built here. to_provider_format( format="openai.responses") is the converter the Responses plugin uses, so the detail level, a url the backend can fetch itself, and the ordering of an image against its caption are decided once rather than twice, and an Instructions part now reaches the caption, which the local version silently dropped. Refs #7291
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…n ends The epoch was advanced once the replacement connected, so everything queued while nothing was connected still carried the dead connection's stamp and was dropped as its work. A frame pushed during a three second reconnect belonged to no session at all, and was thrown away instead of giving the new backend the newest view. It now turns over in the finally that ends a connection, so what is queued in the gap is stamped for whatever comes next. This also leaves the first epoch intact when the opening attempts fail: the finally is only reached once a connection was established, so nothing queued before the first successful connect is mistaken for a previous session's work. Refs #7291
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The voice model has no image channel, so an image goes to the backend Responses model as an input item. Two ways in, both ordinary ones:
Nothing runs on its own: the image waits in the backend's input until the voice model next delegates, or a tool result continues it. Under delegation="client" there is nothing to look at one, so video warns once a session and a context image warns once a message, while its words still reach the voice model.
Queued events now carry the connection they were written for. Audio appends, response.item.create and response.create are dropped rather than replayed into a later session, whose backend has none of the state they name; the backend kinds warn so the caller knows to send them again. Context appends still ride over, since the voice model is just as happy to hear them on the next connection.
addresses #7291