Usage of local models

[flzieg]

Hi, is it possible to use local models that I have downloaded from Hugging Face and how can I load them? Maybe I have overlooked it.

[buhe]

Hi, because the iPhone's RAM is limited, I think we should use the API solution .
Trace -> #86

[flzieg]

Ok, I understand that. But I have a use case where I have to run models on the phone because of security reasons. There is also the app LM Studio that can run local models. I don't know, how they have implemented it, because it is not open source. A collegue has testet this with a Llama 2 7B Q3 model and it was quite fast. But thanks for the fast answer!

[buhe]

Maybe https://tvm.apache.org .

[flzieg]

Ah, it wasn't LM Studio, it was LLMFarm. And LLMFarm is open source. :)

[buhe]

We released 0.46.0, supporting local model!

[flzieg]

Wow, this is sick, thx!

[buhe]

Welcome any PR about it!