[–] [S] 2 points 9 months ago (1 child)

For sure, context rot is a problem, but that's also the easiest thing to control for in this case. If sensor data is relevant to you, having some code to process and reduce it to a dashboard you can read is always a good idea, independently of getting an LLM into the loop.

This becomes more complicated with data you can't really understand like results from blood tests, for example. But maybe you just don't summarize any of that.

  • source
  • parent
  • context
  •  

    Seems to make sense for maximum privacy. Put together a large enough model to answer health queries, have OCR and image recognition to read exams, give it web access to search for medication details and, of course, gather raw data from any devices you use to measure weight, heart rate, etc.

    But that's just in theory and we all know things are hard to put together. In practice, have you had any experience getting anything like this working locally?

     

    Total noob to this space, correct me if I'm wrong. I'm looking at getting new hardware for inference and I'm open to AMD, NVIDIA or even Apple Silicon.

    It feels like consumer hardware comparatively gives you more value generating images than trying to run chatbots. Like, the models you can run at home are just dumb to talk to. But they can generate images of comparable quality to online services if you're willing to wait a bit longer.

    Like, GPT OSS 120b, assuming you can spare 80GB of memory, is still not GPT 5. But Flux Shnell is still Flux Shnel, right? So if diffusion is the thing, NVIDIA wins right now.

    Other options might even be better for other uses, but chatbots are comparatively hard to justify. Maybe for more specific cases like code completion with zero latency or building a voice assistant, I guess.

    Am I too off the mark?

    view more: next ›