You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+17-6
Original file line number
Diff line number
Diff line change
@@ -13,21 +13,32 @@ On October 1st, [OpenAI introduced their Realtime API](https://openai.com/index/
13
13
This (WIP) project integrates the API into a Unity application, allowing users to build low-latency, multi-modal conversational apps that support both text and audio input/output, as well as function calling (via [OpenAI Realtime API documentation](https://platform.openai.com/docs/guides/realtime)).
14
14
15
15
Specifically, this package allows you to integrate real-time low-latency voice conversations with OpenAI's TTS/STT conversational models (via push-to-talk or VAD).
16
+
I've integrated natural speech interruption handling and (a super dumb) client-side VAD.
16
17
17
18
Tested with Unity version 2022.3.45f1 (on macOS, but it should work on every platform that supports `Newtonsoft.Json`).
18
19
19
-
## Demo Video (coming soon)
20
+
## Demos
21
+
> disclaimer, I liked the UI used by OpenAI [in this project](https://github.com/openai/openai-realtime-console/tree/main), as you can see :)
2. Import it into your own project, e.g., via `Assets > Import Package`.
24
-
3. Either open the `DemoScene` scene or add the necessary Prefabs to your own scene.
32
+
3. Go to Package Manager, click on the `+`, then `Add package by name`, and add `com.unity.nuget.newtonsoft-json`.
33
+
4. Either open the `DemoScene` scene or add the necessary Prefabs to your own scene.
25
34
26
35
### Using OpenAI Realtime API in your own scene
27
-
1. Add the `RealtimeAPIWrapper` Prefab to your scene.
28
-
2. Add your OpenAI API key to the `RealtimeAPIWrapper` Prefab.
36
+
1. Add the `RealtimeAPI` Prefab to your scene.
37
+
2. Add your OpenAI API key to the `RealtimeAPI` Prefab.
29
38
3. Optional: Add the `DemoIntegration` Prefab to a Canvas within your scene or open up the `DemoScene` to see an integration example.
30
-
4. Start sending and receiving real-time audio with OpenAI's Realtime API - either by pressing on the button to start and stop recording, or by using the Space-key for push-to-talk.
39
+
4. Reference the `AudioController`, nested inside the `RealtimeAPI` Prefab, and call `audioController.StartRecording()` and `audioController.StopRecording()` to start resp. stop a microphone recording (as push-to-talk).
40
+
5. Bind a button or key to both actions - within my `DemoIntegration`, the spacebar can also be used as push-to-talk key.
41
+
6. The audio is automatically recorded, converted to PCM16 and Base64, sent via the WebSocket connection to the API, and the received chunks and written transcript are handled and served via events.
31
42
32
43
### Action Events
33
44
The `RealtimeAPIWrapper` component triggers various **Action** events, enabling you to handle key stages of the real-time session. Below is a list of available Action events:
0 commit comments