brucethemoose comments on TIL: There is an open source "Alexa replacement" project

TIL: There is an open source "Alexa replacement" project (www.openvoiceos.org)

submitted 11 months ago by cm0002@libretechni.ca to c/linux@programming.dev

38 comments fedilink hide all child comments

As Snowden told us, video and audio recording capabilities of your devices are NSA spying vectors. OSS/Linux is a safeguard against such capabilities. The massive datacenter investments in US will be used to classify us all into a patriotic (for Israel)/Oligarchist social credit score, and every mega tech company can increase profits through NSA cooperation, and are legally obligated to cooperate with all government orders.

Speech to text and speech automation are useful tech, though always listening state sponsored terrorists is a non-NSA targeted path for sweeping future social credit classifications of your past life.

Some small LLMs that can be used for speech to text: https://modal.com/blog/open-source-stt

you are viewing a single comment's thread
view the rest of the comments

[–] brucethemoose@lemmy.world 31 points 11 months ago* (last edited 11 months ago) (14 children)

I mean, there are many. TTS and self-hosted automation are huge in the local LLM scene.

We even have open source "omni" models now, that can ingest and output speech tokens directly (which means they get more semantic understanding from tone and such, they 'choose' the tone to reply with, and that it's streamable word-by-word). They support all sorts of tool calling.

...But they aren't easy to run. It's still in the realm of homelabs with at least an RTX 3060 + hacky python projects.

If you're mad, you can self-host Longcat Omni

https://huggingface.co/meituan-longcat/LongCat-Flash-Omni

And blow Alexa out of the water with a MIT-licensed model from, I kid you not, a Chinese food delivery company.

EDIT

For the curious, see:

Audio-text-to-text (and sometimes TTS): https://huggingface.co/models?pipeline_tag=audio-text-to-text&num_parameters=min%3A6B&sort=modified

TTS: https://huggingface.co/models?pipeline_tag=text-to-speech&num_parameters=min%3A6B&sort=modified

"Anything-to-anything," generally image/video/audio/text -> text/speech: https://huggingface.co/models?pipeline_tag=any-to-any&num_parameters=min%3A6B&sort=modified

Bigger than 6B to exclude toy/test models.

hideshow 14 child comments

[–] fonix232@fedia.io 3 points 11 months ago (13 children)

I do wish there was a smaller LongCat model available. My current AI node has a hard 16GB VRAM limit (yay AMD UMA limitations), so 27B can't really fit. An 8B dynamically loaded model would fit, and run much better.

parent

hideshow 13 child comments

[–] brucethemoose@lemmy.world 3 points 11 months ago* (12 children)

You can do hybrid inference of Qwen 30B omni for sure. Or fully offload inference of Vibevoice Large (9B). Or really a huge array of models.

...The limiting factor is free time, TBH. Just sifting through the sea of models, seeing if they work at all, testing if quantization works and such is a huge timesink, especially if you are trying to load stuff with rocm.

parent

hideshow 12 child comments

[–] fonix232@fedia.io 2 points 11 months ago (11 children)

And I am on ROCm - specifically on an 8945HS, which is advertised as a Ryzen AI APU yet is completely unsupported as a target with major issues around queuing and more complex models (although the new 7.0 betas have been promising but TheRock's flip-flopping with their Docker images has been making me go crazy...).

parent

hideshow 11 child comments

[–] brucethemoose@lemmy.world 2 points 11 months ago* (10 children)

Ah. On an 8000 APU, to be blunt, you're likely better off with Vulkan + whatever omni models GGML supports these days. Last I checked, TG is faster and prompt processing is close to rocm.

...And yeah, that was total misadvertisement on AMD's part. They've completely diluted the term kinda like TV makers did with 'HDR'

parent

hideshow 10 child comments

[–] fonix232@fedia.io 1 point 11 months ago (9 children)

The thing is, if AMD actually added proper support for it, given it has a somewhat powerful NPU as well... For the total TDP of the package it's still one of the best perf per watt APU, just the damn software support isn't there.

Feckin AMD.

parent

hideshow 9 child comments

[–] brucethemoose@lemmy.world 1 point 11 months ago* (8 children)

The IGP is more powerful than the NPU on these things anyway. The NPU us more for 'background' tasks, like Teams audio processing or whatever its used for on Windows.

Yeah, in hindsight, AMD should have tasked (and still should task) a few engineers on popular projects (and pushed NPU support harder), but GGML support is good these days. It's gonna be pretty close to RAM speed-bound for text generation.

parent

hideshow 8 child comments

[–] fonix232@fedia.io 1 point 11 months ago (7 children)

Aye, I was actually hoping to use the NPU for TTS/STT while keeping the LLM systems GPU bound.

parent

hideshow 7 child comments

[–] brucethemoose@lemmy.world 1 point 11 months ago* (6 children)

It still uses memory bandwidth, unfortunately. There's no way around that, though NPU TTS would still be neat.

...Also, generally, STT responses can't be streamed, so you mind as well use the iGPU anyway. TTS can be chunked I guess, but do the major implementations do that?

parent

hideshow 6 child comments

[–] fonix232@fedia.io 2 points 11 months ago (5 children)

Piper does chunking for TTS, and could utilise the NPU with the right drivers.

And the idea of running them on the NPU is not about memory usage but hardware capacity/parallelism. Although I guess it would have some benefits when I don't have to constantly load/unload GPU models.

parent

hideshow 5 child comments

[–] brucethemoose@lemmy.world 2 points 11 months ago (3 children)

Oh, I forgot!

You should check out Lemonade:

https://github.com/lemonade-sdk/lemonade

It's supports Ryzen NPUs via 2 different runtimes... though apparently not the 8000 series yet?

parent

hideshow 3 child comments

[–] fonix232@fedia.io 1 point 11 months ago (2 children)

I've actually been eyeing lemonade, but the lack of Dockerisation is still an issue... guess I'll just DIY it at one point.

parent

hideshow 2 child comments

[–] brucethemoose@lemmy.world 1 point 11 months ago* (1 child)

It's all C++ now, so it doesn't really need docker! I don't use docker for any ML stuff, just pip/uv venvs.

You might consider Arch (dockerless) ROCM soon; it looks like 7.1 is in the staging repo right now.

parent

hideshow 1 child comment

[–] fonix232@fedia.io 2 points 11 months ago

Due to the fact I am running UnRaid on the node in question, I kinda do need Docker. I want to avoid messing with the core OS as much as possible, plus a Dockerised app is always easier to restore.

parent

[–] brucethemoose@lemmy.world 2 points 11 months ago*

Yeah... Even if the LLM is RAM speed constrained, simply using another device to not to interrupt it would be good.

Honestly AMD's software dev efforts are baffling. They've focused on a few on libraries precisely no-one uses, like this: https://github.com/amd/Quark

While ignoring issues holding back entire sectors (like broken flash-attention) with devs screaming about it at the top of their lungs.

Intel suffers from corporate Game of Thrones, but at least they have meaningful contributions in the open source space here, like the SYCL/AMX llama.cpp code or the OpenVINO efforts.

parent