implement the model to reside in memory, rather than reloading the model initialization every time you execute the system command, which is inefficient.

[CementZhang]

Your inference core logic function:

def run_command(command: List[str]) -> str:
"""Run a system command and capture its output."""
try:
result = subprocess.run(command, check=True, capture_output=True, text=True)
return result.stdout
except subprocess.CalledProcessError as e:
raise HTTPException(status_code=500, detail=f"Error occurred while running command: {e}")
except Exception as e:
raise HTTPException(status_code=500, detail=f"Unexpected error: {e}")

This is the same as the way run_inference.pyrun_inference.py executes system commands. Your project simply encapsulates run_command. If you provide API interfaces as a server, you can consider using the resident memory mode instead of loading the model into memory from scratch every time (this is time-consuming and inefficient). For details, you can refer to the "llama.cpp" method and use "llama_cpp.server" to provide services。

[grctest]

Thanks, I'll look to include this fix soon as well as support for the conversational mode required to use Microsoft's latest official BitNet model.

[abstratium-dev]

isn't it just easier to use llama_server, as described here: https://github.com/abstratium-informatique-sarl/bitnet-llm?tab=readme-ov-file

[grctest]

You can now run multiple servers, in the proper server mode.