you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 4 months ago*

Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.

  • source
  • parent