23
I have a RTX A1000 (8GB) with 64GB DRR5, what would you recommend for a newbie? (lemmy.world)
submitted 5 days ago by to c/localllama@sh.itjust.works
32 comments fedilink hide all child comments
 

I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 5 days ago (3 children)

Ok for image generation, I recommend flux 2 klein 4b and run it with stable diffusion which provides both a cli and a web UI. You can also run it through comfyui instead, but after trying it out I wasn't a massive fan.

For the rest I'm confused, you said no to agentic and coding, but then mentioned task logging and summarising documents, which depending on the tasks and documents, can benefit from agentic and coding? You won't be running a sota model, so the llm won't know everything. It will need to be honest about what it does and doesn't know, and be able to research using the internet, then if you want it to produce a document in a folder summarising or reporting for you, this is all agentic stuff, you need to setup MCP tooling.

Genuinely if you want a good general chat bot that can use the internet, if go with lfm2.5 8b a1b on a q4 k m as it will be really fast and good quality. If you want something stronger with the agentic tool calling then orinth 1.5 35b a3b q4 k m.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 3 points 5 days ago* (last edited 5 days ago) (2 children)

    Thanks. Sorry i'm not an expert in AI. No coding meaning i don't intend to do programming, of apps or whatever. No agentic in my limited knowledge is not advanced agents that would do things on its own on conditions, I would always be prompting something. For example for tasks I have a nextcloud server and I use its task app, accessible via caldav for syncing across devices, and I would like the local AI to be able to create a task when prompted, or create a calendar entry. Ideally vocally. But maybe that's already too much?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 5 days ago (1 child)

    Asking an ai to do something for you is essentially agentic usage. All an llm can do is generate text or images or audio or video. The key part is generate. They can't do anything but generate. As soon as you want an llm to do something, you need to give it a tool to do that thing with. It then needs to be capable of working out how to effectively carry out that task.

    The most basic llm will chat to you based on the knowledge it was trained with. Generally, the bigger the llm the more knowledge it has. This is a basic chat bot, the bigger it is the more knowledge it has.

    After this came reasoning and thinking, where the chat bots got smarter. They still had the same amount of knowledge, but they were able to reason, work stuff out, think about things, before answering your question and prompt.

    Then to do stuff, like create docs, search the web, carry out an action, the llm needed to be given tool access and actually be able to do tool calls to use those tools. Anything like create, edit and delete a file, view a folders contents, create a diary entry, search the internet, all required tools. In fact anything except for basic chat response requires a tool. This is where MCP servers and agentic harnesses come in. But you also need an llm capable of tool calling. Giving an llm access to a coding sandbox enables it to be able to calculate answers for you.

    As an example say you want to get an llm to create a diary event for 30 days before Xmas to buy a present for a friend. The llm needs to first work out what day today is, work when Xmas is, then work out what day 30 days before Xmas is, llm's are terrible at maths, but with a coding sandbox, it can create a basic calculation to work out the date to create the diary event for. It would then call a tool to create the diary event on that day.

    Adding voice recognition is another model that takes your voice commands, converts it to text, then passes it to the next llm that does stuff.

    Ai isn't magic. You have to build and create the stack that does what you want. But ai can help you do that.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •