Skip to content

[9.1] [ML][Inference Endpoints] Anthropic endpoint creation: ensure max tokens parameter is passed as expected (#241212) #241352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 30, 2025
Merged

[9.1] [ML][Inference Endpoints] Anthropic endpoint creation: ensure max tokens parameter is passed as expected (#241212) #241352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions x-pack/platform/plugins/shared/inference_endpoint/server/routes/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,18 @@ export const getInferenceServicesRoute = (

const { config, secrets } = request.body;

// NOTE: This is a temporary workaround for anthropic max_tokens handling until the services endpoint is updated to reflect the correct structure.
// Anthropic is unique in that it requires max_tokens to be sent as part of the task_settings instead of the usual service_settings.
// Until the services endpoint is updated to reflect that, there is no way for the form UI to know where to put max_tokens. This can be removed once that update is made.
let taskSettings;
if (config?.provider === 'anthropic' && config?.providerConfig?.max_tokens) {
taskSettings = {
max_tokens: config.providerConfig.max_tokens,
};
// This field is unknown to the anthropic service config, so we remove it
delete config.providerConfig.max_tokens;
}

const serviceSettings = {
...unflattenObject(config?.providerConfig ?? {}),
...unflattenObject(secrets?.providerSecrets ?? {}),
Expand All @@ -117,6 +129,7 @@ export const getInferenceServicesRoute = (
inference_config: {
service: config?.provider,
service_settings: serviceSettings,
...(taskSettings ? { task_settings: taskSettings } : {}),
},
});

Expand Down
11 changes: 10 additions & 1 deletion ..._inference_endpoints/public/components/edit_inference_endpoints/edit_inference_flyout.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import React, { useCallback } from 'react';
import { InferenceEndpoint } from '@kbn/inference-endpoint-ui-common';
import { flattenObject } from '@kbn/object-utils';
import { InferenceInferenceEndpointInfo } from '@elastic/elasticsearch/lib/api/types';
import { ServiceProviderKeys } from '@kbn/inference-endpoint-ui-common';
import { useKibana } from '../../hooks/use_kibana';
import { useQueryInferenceEndpoints } from '../../hooks/use_inference_endpoints';

Expand Down Expand Up @@ -37,7 +38,15 @@ export const EditInferenceFlyout: React.FC<EditInterfaceFlyoutProps> = ({
inferenceId: selectedInferenceEndpoint.inference_id,
taskType: selectedInferenceEndpoint.task_type,
provider: selectedInferenceEndpoint.service,
providerConfig: flattenObject(selectedInferenceEndpoint.service_settings),
providerConfig: {
...flattenObject(selectedInferenceEndpoint.service_settings),
// NOTE: The below is a workaround for anthropic max_tokens handling.
// Anthropic is unique in that it requires max_tokens to be stored as part of the task_settings instead of the usual service_settings - which we populate the providerConfig from.
...(selectedInferenceEndpoint.task_settings?.max_tokens &&
selectedInferenceEndpoint.service === ServiceProviderKeys.anthropic
? { max_tokens: selectedInferenceEndpoint.task_settings?.max_tokens }
: {}),
},
},
secrets: {
providerSecrets: {},
Expand Down