feat: add support for tool_choice to responses api #4106
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
Contributor
✱ Stainless preview buildsThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs. ✅ llama-stack-client-node studio · code · diff
✅ llama-stack-client-kotlin studio · code · diff
✅ llama-stack-client-python studio · code · diff
✅ llama-stack-client-go studio · code · diff
This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push. |
jaideepr97
force-pushed
the
tool-choice
branch
4 times, most recently
from
9bab29b to
55bd671
Compare
jaideepr97
changed the title
jaideepr97
force-pushed
the
tool-choice
branch
2 times, most recently
from
3fd6509 to
a7e1132
Compare
jaideepr97
requested review from
ashwinb,
bbrowning,
ehhuang,
franciscojavierarceo,
hardikjshah,
leseb,
mattf,
raghotham,
reluctantfuturist,
slekkala1,
terrytangyuan and
yanxi0830
as code owners
Signed-off-by: Jaideep Rao <[email protected]>
jaideepr97
force-pushed
the
tool-choice
branch
from
a7e1132 to
09597b6
Compare
ashwinb
reviewed
Nov 13, 2025
| def convert_web_search_tool_choice(chat_tool_names: list[str]) -> dict[str, Any]: | ||
| """Convert a responses tool choice of type web_search to a chat completions compatible function tool choice.""" | ||
| tool_name = "web_search_preview" | ||
| if tool_name not in chat_tool_names: |
Contributor
There was a problem hiding this comment.
This looks for web_search_preview, but _process_new_tools only registers the builtin search tool as web_search, so every tool_choice={"type":"web_search_preview"} hits the ValueError and we fall back to auto. Can we check the internal name (or share the alias list) so web-search tool_choice actually works?
| ) | ||
| continue | ||
| elif tool["type"] == "web_search": | ||
| try: |
Contributor
There was a problem hiding this comment.
Allowed-tools only checks for tool["type"] == "web_search", but the public API sends web_search_preview, web_search_2025_08_26, etc., so those entries drop into the unsupported branch and never constrain the model. Could we treat the WebSearchToolTypes the same way we do when registering the tool?
| top_p: float | None = None | ||
| tools: Sequence[OpenAIResponseTool] | None = None | ||
| tool_choice: OpenAIResponseInputToolChoice | None = None | ||
| truncation: str | None = None |
Contributor
There was a problem hiding this comment.
We add tool_choice to the schema here, but the orchestrator never sets it on OpenAIResponseObject, so stored responses still show null even when the caller specified a tool choice. Can we plumb the requested/processed tool_choice through when building the response?
ashwinb
requested changes
Nov 13, 2025
Contributor
ashwinb
left a comment
ashwinb
left a comment
There was a problem hiding this comment.
A bunch of inline comments. Thanks for this PR!
|
|
||
| async def _process_tool_choice(self) -> str | dict[str, Any] | None: | ||
| """Process and validate the OpenAI Response tool choice and return the appropriate chat completion tool choice string or object.""" | ||
| if self.ctx.responses_tool_choice: |
| tool["function"]["name"] for tool in self.ctx.chat_tools if tool.get("type") == "function" | ||
| ] | ||
|
|
||
| if isinstance(self.ctx.responses_tool_choice, OpenAIResponseInputToolChoiceMode): |
Contributor
There was a problem hiding this comment.
I think you should make _process_tool_choice a free floating function -- there is no reason why it should need "state" from self, whatever 1-2 things it needs can be passed as parameters.
| if isinstance(self.ctx.responses_tool_choice, OpenAIResponseInputToolChoiceMode): | ||
| if self.ctx.responses_tool_choice.value == "required": | ||
| if len(chat_tool_names) == 0: | ||
| logger.warning("No tools available to enforce tool choice, skipping tool choice enforcement") |
Contributor
There was a problem hiding this comment.
I don't think this warning is necessary
| return None | ||
|
|
||
| # add all function tools to the allowed tools list and set mode to required | ||
| return { |
Contributor
There was a problem hiding this comment.
why are we returning untyped dicts? does this correspond to a type somewhere -- feels like it must?
| }, | ||
| } | ||
|
|
||
| elif isinstance(self.ctx.responses_tool_choice, OpenAIResponseInputToolChoiceCustomTool): |
Contributor
There was a problem hiding this comment.
I think we can improve the way this whole function is written perhaps -- these cases feel a bit duplicative? sorry about this being a vague, slightly unactionable comment. but there just feels a bit too much code here for what we want to do.
| return {"type": "function", "function": {"name": "knowledge_search"}} | ||
|
|
||
|
|
||
| def convert_web_search_tool_choice(chat_tool_names: list[str]) -> dict[str, Any]: |
Contributor
There was a problem hiding this comment.
other than the mcp tool choice function, these other things are useless wrappers in my opinion and make the calling code harder to follow actually, what with the try: except: etc. all of that is unnecessary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds support for enforcing tool usage via responses api. See https://platform.openai.com/docs/api-reference/responses/create#responses_create-tool_choice for details from official documentation.
Note: at present this PR only supports
file_searchandweb_search_previewas options to enforce builtin tool usageCloses #3548
Test Plan
uv run --no-sync ./scripts/integration-tests.sh --stack-config server:ci-tests --inference-mode replay --setup ollama --suite base