Add Jvector support to neural-search plugin

[akash-shankaran]

Description

Add support for JVector query engine to neural search plugin. This provides the user an option to use JVector instead of the k-nn plugin.

Related Issues

Resolves #1593

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[heemin32]

Hi. We’re releasing only one flavor of the neural plugin that uses k-NN but not jVector. This means customers who want to use the neural plugin with jVector will have to build the artifact themselves, which isn’t ideal. As a result, this PR may not provide much practical value. What are your thoughts?

[akash-shankaran]

Hi. We’re releasing only one flavor of the neural plugin that uses k-NN but not jVector. This means customers who want to use the neural plugin with jVector will have to build the artifact themselves, which isn’t ideal. As a result, this PR may not provide much practical value. What are your thoughts?

I think you highlight an important gap in the build system and release process, that it will pick one or the other vector engine. Having both in the same release creates a dependency issue. My thought is that coverage (and artifacts) should be present for both engines to be part of the release. We could achieve this by creating:
> Release with 2 flavors of neural plugin - JVector and Knn (default) based.

I propose we break this in 2 steps:

1. The current PR, which allows a user to build Jvector based artifacts. This should have no impact on current release.
2. Next, we modify the neural plugin release to include JVector artifacts. This'll produce a second release, along with the k-nn based release.

WDYT? I'm open to any new ideas as well, which'll reduce maintainence costs of multiple releases. Given the two libraries having dependency issues, I do not see a different way.

[heemin32]

I think it would be better to fork a new repository or create a dedicated branch for jVector until we can eliminate the dependency on k-NN. Since the JVector plugin isn’t part of the OpenSearch distribution artifact, the current approach could significantly increase the neural plugin’s maintenance burden and pose a risk of future compatibility issues between k-NN and jVector.

[akash-shankaran]

I think it would be better to fork a new repository or create a dedicated branch for jVector until we can eliminate the dependency on k-NN. Since the JVector plugin isn’t part of the OpenSearch distribution artifact, the current approach could significantly increase the neural plugin’s maintenance burden and pose a risk of future compatibility issues between k-NN and jVector.

Is k-NN dependency elimination from neural plugin currently on your roadmap? Let me evaluate some of the other alternatives you proposed and circle back.

[heemin32]

I think it would be better to fork a new repository or create a dedicated branch for jVector until we can eliminate the dependency on k-NN. Since the JVector plugin isn’t part of the OpenSearch distribution artifact, the current approach could significantly increase the neural plugin’s maintenance burden and pose a risk of future compatibility issues between k-NN and jVector.

Is k-NN dependency elimination from neural plugin currently on your roadmap? Let me evaluate some of the other alternatives you proposed and circle back.

There are currently no foreseeable plans to remove KNN as a dependency.

[yuye-aws]

Since the JVector plugin isn’t part of the OpenSearch distribution artifact

What's the blocker to include JVector plugin?

From high level, neural-search can depend on both k-NN and jvector plugin. When working on semantic search, users should be able to choose between k-NN and jvector.