Interest in Project 18 (OVMS Embeddings)

[Raghavyadav17]

@dtrawins, @dkalinowski
Dear Dariusz and Damian,

I hope this message finds you both well. My name is Raghav, and I am currently studying Computer Engineering at the University of Toronto. I have had the privilege of engaging in research at the Vector Institute, where my work focused on machine learning applications, model optimization, and AI-based solutions. I am very excited about the opportunity to contribute to the Multimodal Embeddings based on OVMS project, and I wanted to reach out to express my interest and enthusiasm for this initiative.

The idea of developing a system for multimodal embeddings that can seamlessly compare different types of data, such as text, image, audio, and video, aligns perfectly with my academic background and research experience. I have always been fascinated by how AI models can process and make sense of various types of data simultaneously, and this project offers an excellent opportunity to dive deeper into that area. The use of OpenVINO Model Server to create an efficient and scalable solution for serving embeddings is something I find particularly exciting, as it directly relates to challenges I have worked on in my past research, including performance optimization and inference execution.

Specifically, I am very interested in the task of optimizing the preprocessing pipeline and ensuring that data in various formats (images, audio, video) are transformed efficiently for use with AI models. I am also excited about the prospect of integrating the transformers library for preprocessing with OpenVINO Runtime for inference execution. These are tools I have worked with before, and I believe they offer immense potential when combined to create high-performance AI applications.

I would love to connect with you both to discuss the project further and explore how I can contribute. Please let me know if there’s a convenient time for us to have a conversation, or if there are any next steps I should take. I would be writing my finishing my proposal soon, perhaps I can have it reviewed once.

PS: My old github account got suspended for no reason, this is my backup account. I would be adding my projects on this account soon!!!

[Raghavyadav17]

@adrianboguszewski @mlukasze could you please connect me with the mentors, appreciate it !!

[Raghavyadav17]

Subject: Follow-up: Multimodal Embeddings Project Proposal – Awaiting Your Response
@dtrawins, @dkalinowski

Dear Dariusz and Damian,

I hope you’re doing well. I’m following up as I have yet to receive a response to my initial message regarding the Multimodal Embeddings Based on OVMS project. I am eager to finalize my project proposal and need your input to proceed effectively.

Proposed Approaches:

1. Preprocessing Pipeline (Python, LangChain, Transformers)

   • Use LangChain and transformers to handle multimodal data (text, image, audio, video).
   • Standardize inputs to match model requirements (e.g., feature extraction, tensor conversion).
   • Implement a Mediapipe graph for structured preprocessing.

2. Efficient Inference Execution (OpenVINO)

   • Optimize inference with OpenVINO Runtime to ensure low-latency embedding computation.
   • Load and serve transformer-based models from Hugging Face.
   • Deploy dedicated REST API endpoints for multimodal embeddings.

3. Vector Database for Retrieval

   • Store embeddings in a vector database (FAISS, ChromaDB, Weaviate).
   • Implement nearest-neighbor search for semantic similarity.
   • Compare embeddings using cosine similarity or other distance metrics.

4. Sample Application for Demonstration

   • Develop a UI for image search via text prompts.
   • Implement backend retrieval based on stored embeddings.
   • Extend the system to support multimodal retrieval (e.g., video, audio).

Outstanding Questions:

1. Model Selection: Do you have specific models in mind for different modalities, or should we explore various Hugging Face options?
2. Data Pipeline: Should we focus on batch processing or prioritize real-time streaming for scalability?
3. Vector DB Choice: Any preference for FAISS, ChromaDB, or Weaviate, or should we benchmark them?
4. Mediapipe Graph: Are there existing graphs to leverage, or do we need to build from scratch?
5. Performance Benchmarks: Are there efficiency targets (latency, memory usage) we should adhere to?
6. Future Extensibility: Should the system be designed for easy extension to additional modalities?

[dkalinowski]

Hello @Raghavyadav17 we are glad you are interested in contribution to OpenVINO Model Server! 😃

To address your questions:

1. We have no specific models in mind - we should explore current state of the art models for images, video or audio embeddings generation on Hugging Face.
2. OpenVINO batch processing shines when the inputs are of similar sizes, the tensors are of similar shapes. I don't think that our use case will benefit from batching. However, if so - OpenVINO is capable of dynamic batching on backend-level (for devices that really benefit from it, for example Intel GPU), meaning that we might not need to worry about batching on Model Server level. We should of course benchmark it. Regarding streaming, I don't think this is beneficial for embeddings endpoints. We do support text streaming, and new MediaPipe Calculators (which we will end up creating) can also use streaming responses, but I don't think users will benefit from partial embeddings calculations.
3. No preferences for Vector DB choice. However, to clarify - I think the database should be on clientside. OpenVINO Model Server embeddings endpoint should be used to generate embeddings, and the client can store it wherever it wants.
4. OpenVINO Model Server already has text embeddings MediaPipe graph calculator. Please check this demo: https://github.com/openvinotoolkit/model_server/tree/main/demos/embeddings This is very good base for other calculators.
5. There is no KPI as of now, testing Hugging Face, or other embeddings services might be a good starting point. For text embeddings comparisions we have used Infinity serving: https://github.com/michaelfeil/infinity
6. Sure!

Let me know if you have further questions,

Damian

[Raghavyadav17]

@adrianboguszewski would appreciate it if you could connect me to the mentors!!

[Raghavyadav17]

@dkalinowski Thank you so so much for the clarification, I am in the process of completing my proposal. Would it be fine if I could share it with you for you to perhaps review my proposal once ?
Thanks,
Raghav

[NingLi670]

Hi mentors @dtrawins and @dkalinowski ,

My name is Ning Li. I am a first-year Master's student in Computer Science at Shanghai Jiao Tong University. I am very excited about the "Multimodal Embeddings based on OVMS" project. My hands-on experience in multimodal embedding model, dense retrieval, and working with OpenVINO makes me confident in contributing effectively. I would love the opportunity to collaborate and contribute to OpenVINO’s ecosystem through this project.

To gain a deeper understanding of this project for a better application and explore potential extensions, I would like to ask more questions:

1. In addition to text, images, audio, and video, do we need to consider more modalities? For example, structured data (tables, topology graphs, etc.)
2. What's the form of the sample application? A python script, a web page or an desktop application?
3. In order to apply for this project, is there anything you would like us to highlight in the project proposal?

Besides, I'm wondering if I have the opportunity to complete a draft and discuss revisions with you before the official proposal submission date?

Thank you for your time. I look forward to your response and the possibility of contributing to this exciting project.

Best regards,
Ning Li

[dtrawins]

@NingLi670 for the scope of this project I propose to focus on the images, audio and video. In the scope of the sample app, just images would be sufficient but it should be proven that video or audio embedings can be calculated also using appropriate model and appropriate preprocessing from transformers library.
For the sample application I would propose python code of any form. It could be python script with command line, jupyter notebook or gradio UI.

[dtrawins]

@NingLi670 if you would like to consult your proposal before submission you can send it by email.

[Raghavyadav17]

@dtrawins I would like to send my proposal to you before submitting it, would it possible for you to share your email?
Appreciate it,
Thanks!!

[Raghavyadav17]

@dkalinowski perhaps I can get my proposal reviewed by you too, would really appreciate if you could share your email
Thanks

[Raghavyadav17]

@dtrawins @dkalinowski I’ll need to submit my proposal soon—could one of you share your email so I can send it for review? Would appreciate a quick response. Thanks!

[dkalinowski]

@Raghavyadav17 @NingLi670
[email protected]
[email protected]

[Raghavyadav17]

@dkalinowski thank you so much, I will make sure to send my proposal by today for review !!!

[Raghavyadav17]

@dkalinowski @dtrawins i have sent my proposal to you for review, looking forward to your feedback before the submission