3
LLM with Web Search functionality (lemmy.ml)
submitted 4 months ago by to c/selfhost@lemmy.ml
33 comments fedilink hide all child comments
 

Conducting deep web searches and gathering sources is one of the main things I've been using LLMs for. How far away are we from being able to self-host something like Claude's web search capabilities? Or even just a service where I'd pay with my money instead of my data?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 4 months ago* (last edited 4 months ago) (2 children)

Curious why do you swap between Qwen and E4B. On my hardware they perform with similar tps. Qwen 3.6 35B spits out 80-100tps on AMD 9700 and E4B gives me about the same tps.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 4 months ago* (1 child)

    To avoid context switching on the GPU. OpenWebUi for example uses it for memory and title generation.

    Those are not performance critical and background tasks, so instead of slowing down qwen, we just outsource this stuff to the NPU.

    Edit: see here for more details

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •