Skip to content

Commit 235ece1

Browse files
AmberSahdevAmberSahdev
committed
Fixed UI not processing updates, added queue to be processed on the main thread
1 parent ef2913f commit 235ece1

File tree

3 files changed

+114
-12
lines changed

3 files changed

+114
-12
lines changed

‎app/resources/context.txt

Lines changed: 88 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Context:
2-
You are the backend for an app controlling a user's computer. User requests will be conversational such as "Open Sublime text", or "Create an Excel sheet with a meal plan for the week", or "how old is Steve Carrel".
2+
You are Open Interface, the backend for an app controlling a user's computer. User requests will be conversational such as "Open Sublime text", or "Create an Excel sheet with a meal plan for the week", or "how old is Steve Carrel".
33
You return steps to navigate to the correct application, get to the correct text box if needed, and deliver the objective being asked of you as if you were a personal assistant controlling the computer.
44

55
Do this by returning valid JSON responses that map to function calls that can control the mouse, keyboard, and wait (for applications to load) as needed. Only send me back a valid JSON response that I can put in json.loads() without an error - this is extremely important. Do not add any leading or trailing characters.
@@ -61,3 +61,90 @@ Critical guidelines based on your past behavior to help you in your objectives:
6161

6262
Lastly, do not ever, ever do anything to hurt the user or the computer system - do not perform risky deletes, or any other similar actions.
6363
And always reply in valid JSON.
64+
65+
---
66+
pyautogui keyboard documentation
67+
The write() Function
68+
===
69+
The primary keyboard function is ``write()``. This function will type the characters in the string that is passed. To add a delay interval in between pressing each character key, pass an int or float for the ``interval`` keyword argument.
70+
For example:
71+
.. code:: python
72+
>>> pyautogui.write('Hello world!') # prints out "Hello world!" instantly
73+
>>> pyautogui.write('Hello world!', interval=0.25) # prints out "Hello world!" with a quarter second delay after each character
74+
You can only press single-character keys with ``write()``, so you can't press the Shift or F1 keys, for example.
75+
The press(), keyDown(), and keyUp() Functions
76+
===
77+
To press these keys, call the ``press()`` function and pass it a string from the ``pyautogui.KEYBOARD_KEYS`` such as ``enter``, ``esc``, ``f1``. See `KEYBOARD_KEYS`_.
78+
For example:
79+
.. code:: python
80+
>>> pyautogui.press('enter') # press the Enter key
81+
>>> pyautogui.press('f1') # press the F1 key
82+
>>> pyautogui.press('left') # press the left arrow key
83+
The ``press()`` function is really just a wrapper for the ``keyDown()`` and ``keyUp()`` functions, which simulate pressing a key down and then releasing it up. These functions can be called by themselves. For example, to press the left arrow key three times while holding down the Shift key, call the following:
84+
.. code:: python
85+
>>> pyautogui.keyDown('shift') # hold down the shift key
86+
>>> pyautogui.press('left') # press the left arrow key
87+
>>> pyautogui.press('left') # press the left arrow key
88+
>>> pyautogui.press('left') # press the left arrow key
89+
>>> pyautogui.keyUp('shift') # release the shift key
90+
To press multiple keys similar to what ``write()`` does, pass a list of strings to ``press()``. For example:
91+
.. code:: python
92+
>>> pyautogui.press(['left', 'left', 'left'])
93+
Or you can set how many presses `left`:
94+
.. code:: python
95+
>>> pyautogui.press('left', presses=3)
96+
To add a delay interval in between each press, pass an int or float for the ``interval`` keyword argument.
97+
The hold() Context Manager
98+
===
99+
To make holding a key convenient, the ``hold()`` function can be used as a context manager and passed a string from the ``pyautogui.KEYBOARD_KEYS`` such as ``shift``, ``ctrl``, ``alt``, and this key will be held for the duration of the ``with`` context block. See `KEYBOARD_KEYS`_.
100+
.. code:: python
101+
>>> with pyautogui.hold('shift'):
102+
pyautogui.press(['left', 'left', 'left'])
103+
. . .is equivalent to this code:
104+
.. code:: python
105+
>>> pyautogui.keyDown('shift') # hold down the shift key
106+
>>> pyautogui.press('left') # press the left arrow key
107+
>>> pyautogui.press('left') # press the left arrow key
108+
>>> pyautogui.press('left') # press the left arrow key
109+
>>> pyautogui.keyUp('shift') # release the shift key
110+
The hotkey() Function
111+
===
112+
To make pressing hotkeys or keyboard shortcuts convenient, the ``hotkey()`` can be passed several key strings which will be pressed down in order, and then released in reverse order. This code:
113+
.. code:: python
114+
>>> pyautogui.hotkey('ctrl', 'shift', 'esc')
115+
. . .is equivalent to this code:
116+
.. code:: python
117+
>>> pyautogui.keyDown('ctrl')
118+
>>> pyautogui.keyDown('shift')
119+
>>> pyautogui.keyDown('esc')
120+
>>> pyautogui.keyUp('esc')
121+
>>> pyautogui.keyUp('shift')
122+
>>> pyautogui.keyUp('ctrl')
123+
To add a delay interval in between each press, pass an int or float for the ``interval`` keyword argument.
124+
125+
KEYBOARD_KEYS
126+
===
127+
The following are the valid strings to pass to the ``press()``, ``keyDown()``, ``keyUp()``, and ``hotkey()`` functions:
128+
.. code:: python
129+
['\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(',
130+
')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7',
131+
'8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`',
132+
'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o',
133+
'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~',
134+
'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace',
135+
'browserback', 'browserfavorites', 'browserforward', 'browserhome',
136+
'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear',
137+
'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete',
138+
'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10',
139+
'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20',
140+
'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9',
141+
'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja',
142+
'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail',
143+
'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack',
144+
'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6',
145+
'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn',
146+
'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn',
147+
'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator',
148+
'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab',
149+
'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen',
150+
'command', 'option', 'optionleft', 'optionright']

‎app/ui.py

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -240,9 +240,6 @@ def open_advanced_settings(self):
240240
UI.AdvancedSettingsWindow(self)
241241

242242
class MainWindow(ttk.Window):
243-
def change_theme(self, theme_name: str) -> None:
244-
self.style.theme_use(theme_name)
245-
246243
def __init__(self):
247244
settings = Settings()
248245
settings_dict = settings.get_dict()
@@ -274,13 +271,23 @@ def __init__(self):
274271
# This adds app icon in linux which pyinstaller can't
275272
self.tk.call('wm', 'iconphoto', self._w, self.logo_img)
276273

274+
###
277275
# MP Queue to facilitate communication between UI and Core.
278276
# Put user requests received from UI text box into this queue which will then be dequeued in App to be sent
279277
# to core.
280278
self.user_request_queue = Queue()
281279

280+
# Put messages to display on the UI here so we can dequeue them in the main thread
281+
self.message_display_queue = Queue()
282+
# Set up periodic UI processing
283+
self.after(200, self.process_message_display_queue)
284+
###
285+
282286
self.create_widgets()
283287

288+
def change_theme(self, theme_name: str) -> None:
289+
self.style.theme_use(theme_name)
290+
284291
def create_widgets(self) -> None:
285292
# Creates and arranges the UI elements
286293
# Frame
@@ -396,14 +403,17 @@ def update_message(self, message: str) -> None:
396403
if threading.current_thread() is threading.main_thread():
397404
self.message_display['text'] = message
398405
else:
399-
print("calling lambda 1")
400-
self.message_display.after(0, lambda: self._update_message_on_main_thread(message))
401-
print("calling lambda 2")
406+
self.message_display_queue.put(message)
402407
except Exception as e:
403408
print(f"Error updating message: {e}")
404409

405-
def _update_message_on_main_thread(self, message: str) -> None:
406-
# TODO: This lambda is not being executed, investigate
407-
print(f"inside lambda to update message 1 = message: {message}")
408-
self.message_display.config(text=message)
409-
print("inside lambda to update message 2")
410+
def process_message_display_queue(self):
411+
try:
412+
while not self.message_display_queue.empty():
413+
message = self.message_display_queue.get_nowait()
414+
self.message_display.config(text=message)
415+
except Exception as e:
416+
print(f"Error processing message_display_queue: {e}")
417+
418+
# Call this function every 100ms
419+
self.after(200, self.process_message_display_queue)

‎build.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,11 @@ def compile(signing_key=None):
100100
'--hidden-import=ttkbootstrap',
101101
'--hidden-import=openai',
102102

103+
# pypi google_genai doesn't play nice with pyinstaller without this
104+
'--hidden-import=google_genai',
105+
'--hidden-import=google',
106+
'--hidden-import=google.genai',
107+
103108
# NOTE: speech_recognition is the name of the directory that this package is in within ../site-packages/,
104109
# whereas the pypi name is SpeechRecognition (pip install SpeechRecognition).
105110
# This was hard to pin down and took a long time to debug.

0 commit comments

Comments
 (0)