The main script here is ~/generate_commit_message.sh, which uses llama2-8b merged with a finetuned model JosineyJr/generate-conventional-commit-messages to read staged changes (or unstaged if none exist), generate a commit message, and then commit&push it with a fun, emoji-packed CLI.
It assumes that you have the model and compiled llama.cpp downloaded. if you don't (they're not tracked by git so if you're just pulling the repo then you won't); the scripts I wrote to download & compile are in:
- ~/building_model/ and
- ~/building_model/download_model_and_minimal_test/
You'll very likely have to write your own model compilation toolchain, unless you're running NixOS, in which case you can cd into ~/nix-shells/c-nix-shell, run $ nix-shell, and happily continue on your journey with the C compiler that I was using!
Speaking of ~/nix-shells, you can equally cd into ~/nix-shells/python_nix_shell, run $ nix-shell, and happilr continue on your journey with the python .venv that I was using. Are you eyebrows raised yet? I just promised you .venv consistency without downlaoding a .venv! This is unheard of among python programmers, who's solution to package management has traditionally been to say "it works on my machine".
My takeaway from this project is that yes, I (and you!) can run an llm locally using just CPU and RAM. That said, the commits generated by the model run locally kinda suck--API model use from a major provider like OpenAI, Anthropic, etc. will always be better. The biggest gain in my productivity doing this came from environment control. NixOS let me set project-specific C compiler and python .venv constructors that I could switch into and out of freely, update as needed, and even call from other scripts!
(Imagine you're writing a python script to run your model, discover you forgot to compile it, and decide to fix that by writing code in the very same python script that effectively says "become a C terminal temporarily, compile the model, and then take off your dirty C compiler jacket so we can keep being a python script the whole time.")
You can go thru my (repo)[https://github.com/mckinlde/local_llms] if you want, but honestly I don't reccomend it. If you want to compile your own model locally you're better off starting from scratch, and if you want to get the productivity gain I got from doing this you're better off trying out (NixOS)[https://nixos.org/], and I reccomend (these)[https://youtu.be/CwfKlX3rA6E?si=0vusOV5zdaBHXkn7] (intro)[https://youtu.be/9OMDnZWXjn4?si=FJoNLK2z2BEjgZon] (videos)[https://youtu.be/61wGzIv12Ds?si=lG0eqpFtBj5mQCte]
git clone https://github.com/ggml-org/llama.cpp.git
[dmei@nixos:~/experiments/local_llms]$ cd nix-shells/
[dmei@nixos:~/experiments/local_llms/nix-shells]$ tree . ├── c-nix-shell │ └── shell.nix └── python-nix-shell └── shell.nix
3 directories, 2 files
[dmei@nixos:~/experiments/local_llms/proven_scripts]$ tree . ├── convert_merged_model_to_gguf.sh └── merge_lora.py
1 directory, 2 files
👏 Done 🕵️ verify existence -rw-r--r-- 1 dmei users 8.0G Jun 2 22:06 /home/dmei/experiments/local_llms/merged-model/merged_model.gguf
[dmei@nixos:~/experiments/local_llms/llama.cpp]$ cd ../../../git_lfs_models/
[dmei@nixos:~/git_lfs_models]$ tree . ├── generate-conventional-commit-messages │ ├── adapter_config.json │ ├── adapter_model.safetensors │ └── README.md └── Meta-Llama-Guard-2-8B ├── config.json ├── generation_config.json ├── LICENSE ├── model-00001-of-00004.safetensors ├── model-00002-of-00004.safetensors ├── model-00003-of-00004.safetensors ├── model-00004-of-00004.safetensors ├── model.safetensors.index.json ├── original │ ├── consolidated.00.pth │ ├── params.json │ └── tokenizer.model ├── README.md ├── special_tokens_map.json ├── tokenizer_config.json ├── tokenizer.json └── USE_POLICY.md
4 directories, 19 files