Skip to content

Behavior when chunk type check not passed #5

Open
@johnd0e

Description

@johnd0e

Consider such (real) api response:

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[],"usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}}

data: [DONE]

The response objects are missing "object": "chat.completion.chunk", so they aren't passing the check here:

lua-openai/openai/init.moon

Lines 253 to 254 in 08128b9

if chunk = parse_completion_chunk cjson.decode json_blob
chunk_callback chunk

This leads to to some consequences:

  1. Json is valid and decodes without problems.
  2. However parse_completion_chunk doesn't accept it.
  3. As a result, it's simply ignored.
  4. And chunk_callback is never invoked.

It might be more sensible to throw an error instead of just disregarding it.
Or alternatively, invoke chunk_callback regardless, but with an additional isInvalid argument.

The latter is more flexible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions