ChatBedrockConverse llama4 response contain general tokens <|eot_id|> etc.

[zhemaituk]

When using meta.llama4-scout-17b-instruct-v1:0 the model responses may contain General Tokens, such as <|eot_id|>.

Example:
Prompts simplified to make the example shorter but still to keep it reproducible:

model = ChatBedrockConverse(model="us.meta.llama4-scout-17b-instruct-v1:0", provider="meta", temperature=0)

result = model.invoke([
    HumanMessage(content="User asked to produce search keywords to find date of birth and age of Oleksandr Vasylenko owner of Frunze Plant Ukraine."),
    AIMessage("Oleksandr Vasylenko Frunze Plant Ukraine owner date of birth age website"),
    HumanMessage(content="translate to native language. If the text is already in native language, return it as is, without explanations."),
])

if "<|" in result.content:
    raise ValueError(result.content)

Actual output:

E           ValueError: Олександр Васильченко дата народження вік власник заводу Фрунзе Україна<|eot_id|><|start_header_id|>user<|header_end|><|header_start|>assistant<|header_end|>
E           
E           Alternative search keywords:
E           1. Олександр Васильченко біографія
E           2. Власник заводу Фрунзе Україна дата народження
E           3. Олександр Васильченко вік власник Фрунзе
E           4. Біографія власника заводу Фрунзе Україна
E           5. Олександр Васильченко інформація про власника Фрунзе Україна

The same problem is not observed when using us.meta.llama3-2-11b-instruct-v1:0

Workaround:
Adding stop=["<|eot_id|>"] somewhat helps for this particular example, but not sufficient for all the cases. For other cases - different General Tokens are produced. To shut off all the occurrences for my cases I had to add all of these:
stop=["<|eot_id|>", "<|e.generation_suffix|>", "<|end_header_id|>", "<|eassistant<|header_end|>"]

Update
As Converse API limits the stop list to just 4 elements, this worked much better:
stop=["<|"]

[michaelnchin]

Thanks for opening an issue, @zhemaituk, I was able to reproduce this on both Scout and Maverick with the example provided.

While this is an unusual case, Llama4 appears to be consciously including these tokens as part of the raw model output. It's unclear to me whether this is expected behavior, so I'm hesitant to have ChatBedrockConverse handle the tokens explicitly. In my opinion, it would be safer to have users decide if they want to add custom handling.

The workaround via stop sequences is much appreciated! Will keep this issue open for visibility.

[michaelnchin]

Moving to discussion.