llama.cpp built for hip/rocm (since I have AMD GPUs) and a bunch of custom code. One of the first things I did with LLMs was make my own chat client. Most of the useful stuff I get out of LLMs right now involves small focused tasks that I run in a batch with custom scripts (e.g. bulk OCR / image analysis tasks and stuff like that).