Open
Description
Consider such (real) api response:
data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}
data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" Hello!"},"finish_reason":null}]}
data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[],"usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}}
data: [DONE]
The response objects are missing "object": "chat.completion.chunk"
, so they aren't passing the check here:
Lines 253 to 254 in 08128b9
This leads to to some consequences:
- Json is valid and decodes without problems.
- However
parse_completion_chunk
doesn't accept it. - As a result, it's simply ignored.
- And
chunk_callback
is never invoked.
It might be more sensible to throw an error instead of just disregarding it.
Or alternatively, invoke chunk_callback
regardless, but with an additional isInvalid
argument.
The latter is more flexible.
Metadata
Metadata
Assignees
Labels
No labels