Reaching out to understand Project scope for "Gesture Control with OpenVINO"

[AnuragKurle]

Hi @openvino-dev-samples and (couldn't find exact @ for Zhuo Wu),

Very interesting project, but would like to understand more about it, since it was not explicitly mentioned.

For example - hand based gestures might be useful when (say) browsing photos [zooming/ swiping] , reading a blog [scrolling], or playing stone papers scissors [lookup of gesture/ action pairs] - What i mean to say is we can think of many applications and use cases for hand gestures and their control parameters - but if you could maybe share your idea for this "control system" and its core functionalities/ expectations - that would make scoping out this application and define the control system, and thinking about approaching this much clearer and streamlined.

Also let me know in case this would be entirely up to the candidates too!

Thanks!
Anurag

[openvino-dev-samples]

Hi @AnuragKurle, thanks for your attention.

In my mind, porting Mediapipe's example to OpenVINO is the key and first step in this project. you can find the this example as the link.
The second stage is totally up to your design, whatever you think is useful to build with this basic example is OK.

[Hmm-1224]

Hi mentors,

I have started working on the Gesture Control with OpenVINO project and have made progress in implementing real-time gesture detection using MediaPipe landmarks and the JavaScript webcam. Here’s my current approach:

1.Model Conversion and Setup: I have successfully converted the MediaPipe Gesture Recognition model to OpenVINO format and compiled the model for inference.
2.Gesture Recognition: I have defined specific gestures using MediaPipe landmarks for recognition.
3.Real-time Detection: I am capturing gestures via JavaScript webcam for real-time processing.
4.Integration with Flask: The detected landmark data is sent to the Flask backend, which compares it with predefined gestures and determines the corresponding action.
5.Monitor Control Attempt with X11: I am working on implementing monitor control (e.g., brightness adjustment, display switching) using X11 on Linux.

However, while working with X11, I am encountering the following error:
Xlib.xauth: warning, no xauthority details available

Although I am working in the display environment, it seems related to xauth permissions or DISPLAY environment settings. Could you provide any guidance or resources on how to resolve this error and effectively interact with X11 for monitor control?

Looking forward to your feedback and suggestions!

Thanks!
Sonal kumari

[openvino-dev-samples]

Hi @Hmm-1224

Your approach looks good to me.
For 5th approach, I believe these are more like a detailed tech issue during researching, and we can discuss on it once your proposal is accepted.
How about let's focus on your idea instead of implementation details at this stage ?

[Hmm-1224]

Thank you for your feedback! That makes sense—I agree that we can focus on the core idea for now and revisit the technical details once the proposal is accepted.

Core Idea of the Project:
1.Gesture Recognition with OpenVINO

• Convert and optimize MediaPipe’s Gesture Recognizer model for OpenVINO to run efficiently on local hardware (CPU, GPU, VPU).
• Ensure real-time gesture detection with minimal latency

2.Hand Tracking & Landmark Detection

• Detect hands and extract key landmarks using OpenVINO-optimized models.
• Map recognized gestures to predefined actions (e.g., swipe left to switch applications, thumbs up to confirm actions).

3.Local Execution

• Unlike cloud-based solutions, this system will run entirely on local hardware, ensuring privacy, low latency, and offline usability.

4.Customizable Model & Gesture Mapping
-Allow users to define custom gestures and map them to specific application features or commands.

5.Application Integration

• Develop an API that integrates gesture control into various applications (e.g., media players, presentations, gaming, accessibility tools).

Use Cases:
1.Touchless Media Control

• Use hand gestures to play, pause, adjust volume, or skip tracks in media players like VLC or Spotify.

2.Presentation Control

• Swipe left or right to change slides in PowerPoint or Google Slides.
• Raise a palm to start/stop the presentation without touching the keyboard.

3.Web Browsing & Application Navigation

• Use gestures to scroll, switch tabs, or go back/forward in a browser.
• Open applications or navigate system settings hands-free.

4.Gaming & Virtual Reality

• Enhance PC gaming with gesture-based actions for interactive gameplay.
• Accessibility for Physically Impaired Users.(eg:-Moving a hand over different sections of the virtual keyboard to selects keys.)

5.Enable users with limited mobility to control their computer using hand gestures.

• Replace mouse and keyboard inputs with an intuitive, touchless system.

6.Smart Home Integration

• Control IoT devices (e.g., adjust lights, control smart TVs, or manage appliances) using simple hand gestures.

[Hmm-1224]

hello @openvino-dev-samples,
I hope you are doing well. I had sent you my proposal for this idea through mail and you ask me about if i am planning to implement all use cases and specific ones? So, for now i am particularly focused on first two usecases. If you have any question regarding this and any suggestion then please do tell me...I am attaching the proposal below. Please have a look on those two usecases as i have make changes in synopsis part like i added some code part for better understanding..
I know this period will be so busy for you, but i would appreciate if you take out some time to review this.
waiting for your response!!
OpenVINO.pdf

[openvino-dev-samples]

Hi @Hmm-1224 Thanks for your effects.

Could you help to summarize the outcome of this project you can commit with highlighted bullet points? For example :

A gesture control system support:
•	…
•	…
•	…

Thanks

[Hmm-1224]

hello @openvino-dev-samples,
Thank you so much for taking out your precious time in reviewing it..
A gesture Control System Support:

• Real-time Gesture Recognition: Detects and classifies hand gestures using OpenVINO-optimized models.
• MediaPipe to OpenVINO Migration: Successfully porting the MediaPipe Gesture Recognizer to OpenVINO for efficient inference.
• Optimized Performance: Faster and lightweight gesture recognition with improved inference speed on edge devices.
• Application Integration: Enabling gesture-based interaction with applications (e.g., controlling a monitor, media player, or UI navigation).
• Customizable Gesture Models: Fine-tuning and extending gesture recognition capabilities based on task-specific requirements.
• Broad Hardware Support: Optimized for efficient execution on Intel CPUs, GPUs, and VPUs.
• WebSocket API Support: Enables real-time communication between the gesture control system and external applications via WebSocket API.

please have a look on these outcomes and share your suggestions.
waiting for your reply!!

[Hmm-1224]

Hello @zhuo-yoyowz,
I want your review as well for my proposal and i welcome your suggestions and will work on it for improvement. Also Ethan sir had asked questions after reviewing, i want you to please have a look on it as well.. Attaching my proposal below:
OpenVINO.pdf
Waiting for your reply!!

[Hmm-1224]

Hello @openvino-dev-samples and @zhuo-yoyowz,
i request both of you to please check out my update as we have very less time left. If there is any suggestions i am open to take it and work on it..
I also did PR for implementing argmin and argmax using openvino backend and its almost done...
u can find the PR below:
keras-team/keras#21060
I am also working on hstack implemntation when argmin and argmax get merged successfully i will complete hstack task as well...
i also did PR for dot implementation but had to close it cause other contributors did merge it succesfully..
waiting for your response...

[openvino-dev-samples]

Hi Since there are many proposal review requests, I dont have bandwidth to give detailed feedback to each of them.
Generally your outcome of project looks good to me. I suggest you to focus on one application integration use case and optimize the UX for it specifically.

[vijaykr338]

Hi @openvino-dev-samples,

I hope this message finds you well. I had emailed my revised proposal for the "Gesture Control with OpenVINO" project earlier this week and would greatly appreciate your review when you have a moment. I understand it's a really busy time with the deadline in a few days, any feedback would be immensely valuable for finalizing my submission.

I'm pleased to share that one of my PRs has been successfully merged
PR
and I'm currently working on additional contributions to further enhance the PyTorch front-end functionality.

Thank you for your time and consideration.

Best regards,
Vijay Kumar
[[email protected]]

[Hmm-1224]

Hey @openvino-dev-samples ,
Thanks a lot for your valuable time and feedback! I'll focus on refining the UX for a specific application integration use case as you suggested.

[Hmm-1224]

Hello @openvino-dev-samples,

I have submitted my proposal on the GSoC website and am truly excited about the opportunity to work with the OpenVINO team. Looking forward to work with You ......

Best regards,
Sonal Kumari

[Hmm-1224]

hello @openvino-dev-samples ,
just to make sure i submitted correctly, proposal should be submitted at below link right?
https://summerofcode.withgoogle.com/programs/2025/organizations/openvino-toolkit/create-proposal