[Contribution Idea] Support OpenVINO as a backend in Keras 3

[rkazants]

Dear Keras team, @fchollet,

My name is Roman Kazantsev and I work as an AI frameworks engineer with the OpenVINO team at Intel. OpenVINO is a popular solution from Intel for optimizing AI inference.

We like the idea of multi-backend functionality offered by Keras 3, which allows users to easily switch between different deep learning frameworks (JAX, PyTorch, and TensorFlow) for the same model. This is very cool!

How about collaborating to cover more DL frameworks? For example, the OpenVINO team would like to contribute to Keras 3 by adding OpenVINO as a new backend to support inference. We are ready to implement and maintain this feature. Please share your thoughts on this.

Best regards,
Roman

[fchollet]

Hi Roman,

Absolutely, if you want to work on an OpenVINO backend we'd be happy to support you. We don't really bandwidth to actively develop new backends (e.g. our MLX backend is not making progress due to lack of resources) but we're able to answer your questions and review any PRs you submit.

adding OpenVINO as a new backend to support inference

Does OpenVINO not support gradients? Or is it otherwise not a good fit for training?

[rkazants]

Does OpenVINO not support gradients? Or is it otherwise not a good fit for training?

Hi François,

Thanks a lot for your response. OpenVINO provides functionality only for inference and its optimization using compression and quantization. So the plan is to support Keras API for inference in case OpenVINO backend. Don't you mind us to add OpenVINO due to this "limitation"?

I see that we need to add openvino directory into backends with implemented helpers for Keras op decompositions using OpenVINO opset. Anything else can you recommend to take a look? Or other useful developer documentation?
Now we plan to estimate required work, implement protype to infer some basic network and get back.

Will be grateful to you for further response.

Best regards,
Roman

[fchollet]

So the plan is to support Keras API for inference in case OpenVINO backend. Don't you mind us to add OpenVINO due to this "limitation"?

We already have a backend that is inference-only, the NumPy backend. This is not a big issue.

I see that we need to add openvino directory into backends with implemented helpers for Keras op decompositions using OpenVINO opset. Anything else can you recommend to take a look? Or other useful developer documentation?

Since you're only targeting inference, there's a lot you won't need to implement -- no Trainer class, no optimizers. I suggest you look the most closely at what is implemented in keras/backend/numpy/, since that's the other inference-only backend. If something is confusing, you can look at how it's implemented in other backends -- you have 3 other reference implementations for each function.

I think the main point of difficulty might be static shape inference (compute_output_spec), depending on how OpenVINO works.

Now we plan to estimate required work, implement protype to infer some basic network and get back.

Great -- don't hesitate to reach out with questions!

[dhantule]

Hi @rkazants, Thanks for reporting this. Is this issue resolved now?

[rkazants]

Hi @dhantule,

I think - yes. The core part was done here: #19727.
Now we are working on covering more operations: #20910

[dhantule]

Hi @rkazants, please close the issue if it's resolved. Thanks!