mtmd : support home-cooked Mistral Small Omni

[ngxson]

Support a home-cooked version of Mistral Small which can take both audio and image as input

Link to GGUF: https://huggingface.co/ngxson/Home-Cook-Mistral-Small-Omni-24B-2507-GGUF

Try it

# audio
llama-mtmd-cli \
  -m ../models/mistral_omni_gguf/Home-Cook-Mistral-Small-Omni-2507-Q4_K_M.gguf \
  --mmproj ../models/mistral_omni_gguf/mmproj-model-f16.gguf \
  --audio ../models/i-have-a-dream-30s.mp3  \
  -p "Transcribe this"

# Here is the transcription of the provided text:
#
# "I have a dream that one day every valley shall be exalted, every hill and mountain shall be made low, the rough places will
# be made plain, and the crooked places will be made straight, and the glory of the Lord shall be revealed, and all flesh shall
# see it together. This is our hope. With this faith, we will be able to hew out of the mountain of despair a stone of hope."


# vision
llama-mtmd-cli \
  -m ../models/mistral_omni_gguf/Home-Cook-Mistral-Small-Omni-2507-Q4_K_M.gguf \
  --mmproj ../models/mistral_omni_gguf/mmproj-model-f16.gguf \
  --image ../models/bliss.png  \
  -p "What is this"

# This image depicts a serene and expansive landscape featuring a vast, rolling green field under a clear blue sky dotted with 
# fluffy white clouds. The field appears to be lush and well-maintained, possibly a meadow or pasture. The horizon is marked by 
# a gentle, grassy hill that adds depth and dimension to the scene. The overall atmosphere is tranquil and inviting, 
# suggesting a peaceful countryside setting.

[CISC]

Very cool! :)

[stduhpf]

Ah I was trying to make something like that work too. Nice job

[stduhpf]

Maybe the merge_mmproj_models.py script should be included in llama.cpp? (and maybe edited to be able to support other architectures like qwen2.5-omni ?)

[ngxson]

merge_mmproj_models.py is quite a one-off solution for now, so I don't think it's useful to be included inside llama.cpp. It's needed because the original model are literally 2 different models.

We don't need that for qwen2.5-omni, because both audio+vision tensors are all included in the original model.