Chat completion endpoint #33

amannaik247 · 2025-09-17T04:35:07Z

This endpoint will serve the conversation part of the story builder feature in the write activity. As well as a general endpoint which can be utilized to generate conversational responses from the gemma model.

Let's say we are using a gemma model.

The gemma models are trained using specific tokens to generate model responses using a chat history. More can be understood in this documentation.
The main purpose of the endpoint is to take input of the chat history as a list(along with the system prompt in it), like this one:

"messages": [
    {"role": "system", "content": "You are a helpful AI assistant.Reply in 2-3 sentences."},
    {"role": "user", "content": "What is machine learning?"},
    {"role": "assistant", "content": "Machine learning is a subset of artificial intelligence..."},
    {"role": "user", "content": "Can you give me a simple example?"}
  ]

Then convert it to LLM compatible format like this one for gemma:

<start_of_turn>user
You are a helpful AI assistant.Reply in 2-3 sentences.\n What is machine learning?<end_of_turn>
<start_of_turn>model
Machine learning is a subset of artificial intelligence...<end_of_turn>
<start_of_turn>user
Can you give me a simple example?<end_of_turn>

Using this to generate a reasonable model response.
This endpoint can be used by any other future activities which want to introduce a conversation related feature as well. Since the endpoint is similar to most popular chat completion endpoints, integration with sugar-ai will become easier.

PS- I tested this locally using a gemma-3-1b-it and Qwen-2-1.5B-Instruct models.

- Provide a chat-style completions API that accepts multi-turn context (system, user, assistant) and returns a single assistant message. - Normalizes roles (merge first system into first user, map assistant→model) and builds prompts via tokenizer chat template. This is to make it compatible for gemma based models. - Returns choices[0].message with quota metadata; additive, no breaking changes.

Added usage details about the endpoint similar to other existing endpoints

…tions Handle system message placement based on conversation starter: - Merge system into first user message when conversation starts with user - Create separate user message for system content when starting with assistant - Simplify logic by eliminating redundant loops and state tracking

amannaik247 · 2025-09-19T13:37:23Z

@chimosky @MostlyKIGuess Please review and let me know if there are any changes needed.

app/ai.py

app/routes/api.py

chimosky · 2025-09-23T22:22:08Z

Everyone adding an endpoint for something new isn't a good thing moving forward, this would end up becoming bloated.

@MostlyKIGuess We need to figure out a better way to handle this.

amannaik247 · 2025-09-25T12:45:46Z

@chimosky made all the necessary changes and added a refactor commit.

chimosky · 2025-09-25T21:31:29Z

app/ai.py

        except Exception as e:
-            raise Exception(f"Error generating chat completion: {str(e)}")
+            raise Exception(f"Error generating chat completion: {str(e)}")
+


Another no new line at EOF, you should view your changes with git diff before making a commit as you'd easily spot these if you did.

Thank you for the advice . Will keep this in check the next time!

MostlyKIGuess · 2025-09-26T10:51:00Z

Everyone adding an endpoint for something new isn't a good thing moving forward, this would end up becoming bloated.

@MostlyKIGuess We need to figure out a better way to handle this.

I think @mebinthattil 's custom end point should be perfect for everyone.

@mebinthattil Could you share maybe a documentation on it on the website? That would be a great PR addition

MostlyKIGuess · 2025-09-26T10:48:52Z

app/ai.py

            raise Exception(f"Error generating response with custom prompt: {str(e)}")

-    def _normalize_chat_messages(self, messages: list) -> list:
+    def _normalize_chat_messages(self, messages: list[dict]) -> list[dict]:


Any reason why?

Ibiam suggested that it would be better to mention that it is a list of dictionary.

MostlyKIGuess · 2025-09-26T10:49:12Z

app/routes/api.py

Only changes on api.py are just spacing, makes a bit weird to review.. maybe can restore this

mebinthattil · 2025-09-26T11:13:49Z

Everyone adding an endpoint for something new isn't a good thing moving forward, this would end up becoming bloated.
@MostlyKIGuess We need to figure out a better way to handle this.

I think @mebinthattil 's custom end point should be perfect for everyone.

@mebinthattil Could you share maybe a documentation on it on the website? That would be a great PR addition

Yes, the whole point of #29 was so that every activity would not need to raise a PR like this for that activity's usecase. I was in a bit of a rush to get that change out and did not document it well. I shall make a comprehensive documentation about it and also update the website regarding the same. Thanks for pointing this out.

@amannaik247 Try checking out how I implemented this in the speak-ai activity using this generic endpoint.

amannaik247 · 2025-09-26T11:16:00Z

I think @mebinthattil 's custom end point should be perfect for everyone.

Yes, the whole point of #29 was so that every activity would not need to raise a PR like this for that activity's usecase. I was in a bit of a rush to get that change out and did not document it well. I shall make a comprehensive documentation about it and also update the website regarding the same. Thanks for pointing this out.

@MostlyKIGuess @mebinthattil i am using his endpoint for one task. But it only accepts a a custom prompt and a question. There is no way to send a chat history and get a conversation style response. So for providing the context of the whole conversation this endpoint is necessary. Most popular API's have this /chat/completions endpoint.

As I mentioned in my initial PR message that all models have a seperate way to generate a response when we send a chat history because it is trained on those specific tokens.

- Made a few changes to make it pep8 compliant. - Used an alternative clear sentence to explain a function.

chimosky · 2025-09-29T14:59:47Z

@MostlyKIGuess @mebinthattil i am using his endpoint for one task. But it only accepts a a custom prompt and a question. There is no way to send a chat history and get a conversation style response. So for providing the context of the whole conversation this endpoint is necessary. Most popular API's have this /chat/completions endpoint.

This makes sense, it'll also be great to modify the endpoint Mebin created to accept this rather than create a whole new endpoint, you can use a default value - set to None - which would have no effect if nothing is passed there.

This would be easier to maintain than a whole new endpoint.

amannaik247 added 4 commits September 17, 2025 09:38

add docs for /chat/completions endpoint

6f06045

Added usage details about the endpoint similar to other existing endpoints

add usage instructions of /chat/completions endpoint to the web page

a2db502

chimosky reviewed Sep 23, 2025

View reviewed changes

app/ai.py Outdated Show resolved Hide resolved

app/ai.py Outdated Show resolved Hide resolved

app/ai.py Outdated Show resolved Hide resolved

app/routes/api.py Outdated Show resolved Hide resolved

chimosky reviewed Sep 25, 2025

View reviewed changes

MostlyKIGuess reviewed Sep 26, 2025

View reviewed changes

refactor: improve code quality - PEP 8, type hints, and docstrings

ada89de

- Made a few changes to make it pep8 compliant. - Used an alternative clear sentence to explain a function.

amannaik247 force-pushed the aman/chat branch from 5b901c5 to ada89de Compare September 26, 2025 12:01

Chat completion endpoint #33

Are you sure you want to change the base?

Chat completion endpoint #33

Uh oh!

Conversation

amannaik247 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amannaik247 commented Sep 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chimosky commented Sep 23, 2025

Uh oh!

amannaik247 commented Sep 25, 2025

Uh oh!

chimosky Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

amannaik247 Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

MostlyKIGuess commented Sep 26, 2025

Uh oh!

MostlyKIGuess Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

amannaik247 Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

MostlyKIGuess Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

mebinthattil commented Sep 26, 2025

Uh oh!

amannaik247 commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chimosky commented Sep 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amannaik247 commented Sep 17, 2025 •

edited

Loading

amannaik247 commented Sep 26, 2025 •

edited

Loading