Skip to content

Commit bdcf929

Browse files
authored
Update README.md
1 parent 32ab331 commit bdcf929

File tree

1 file changed

+17
-6
lines changed

1 file changed

+17
-6
lines changed

README.md

+17-6
Original file line numberDiff line numberDiff line change
@@ -13,21 +13,32 @@ On October 1st, [OpenAI introduced their Realtime API](https://openai.com/index/
1313
This (WIP) project integrates the API into a Unity application, allowing users to build low-latency, multi-modal conversational apps that support both text and audio input/output, as well as function calling (via [OpenAI Realtime API documentation](https://platform.openai.com/docs/guides/realtime)).
1414

1515
Specifically, this package allows you to integrate real-time low-latency voice conversations with OpenAI's TTS/STT conversational models (via push-to-talk or VAD).
16+
I've integrated natural speech interruption handling and (a super dumb) client-side VAD.
1617

1718
Tested with Unity version 2022.3.45f1 (on macOS, but it should work on every platform that supports `Newtonsoft.Json`).
1819

19-
## Demo Video (coming soon)
20+
## Demos
21+
> disclaimer, I liked the UI used by OpenAI [in this project](https://github.com/openai/openai-realtime-console/tree/main), as you can see :)
22+
### Standard conversation
23+
https://github.com/user-attachments/assets/73b69bd7-dae8-4d49-8f7f-02fa32d9a955
2024

21-
### Setup (coming soon)
25+
### Interruption example
26+
https://github.com/user-attachments/assets/88fd83eb-e285-488d-8308-a40a48a7307f
27+
28+
29+
### Setup
2230
1. Download the latest release `.unitypackage`.
2331
2. Import it into your own project, e.g., via `Assets > Import Package`.
24-
3. Either open the `DemoScene` scene or add the necessary Prefabs to your own scene.
32+
3. Go to Package Manager, click on the `+`, then `Add package by name`, and add `com.unity.nuget.newtonsoft-json`.
33+
4. Either open the `DemoScene` scene or add the necessary Prefabs to your own scene.
2534

2635
### Using OpenAI Realtime API in your own scene
27-
1. Add the `RealtimeAPIWrapper` Prefab to your scene.
28-
2. Add your OpenAI API key to the `RealtimeAPIWrapper` Prefab.
36+
1. Add the `RealtimeAPI` Prefab to your scene.
37+
2. Add your OpenAI API key to the `RealtimeAPI` Prefab.
2938
3. Optional: Add the `DemoIntegration` Prefab to a Canvas within your scene or open up the `DemoScene` to see an integration example.
30-
4. Start sending and receiving real-time audio with OpenAI's Realtime API - either by pressing on the button to start and stop recording, or by using the Space-key for push-to-talk.
39+
4. Reference the `AudioController`, nested inside the `RealtimeAPI` Prefab, and call `audioController.StartRecording()` and `audioController.StopRecording()` to start resp. stop a microphone recording (as push-to-talk).
40+
5. Bind a button or key to both actions - within my `DemoIntegration`, the spacebar can also be used as push-to-talk key.
41+
6. The audio is automatically recorded, converted to PCM16 and Base64, sent via the WebSocket connection to the API, and the received chunks and written transcript are handled and served via events.
3142

3243
### Action Events
3344
The `RealtimeAPIWrapper` component triggers various **Action** events, enabling you to handle key stages of the real-time session. Below is a list of available Action events:

0 commit comments

Comments
 (0)