NLnet grant announcement: https://github.com/Helium314/HeliBoard/issues/2226
NLnet project description: https://nlnet.nl/project/GestureTyping/

Swipe-o-Scope repository: https://codeberg.org/eclexic/swipe-o-scope

CC BY-SA 4.0 license: https://creativecommons.org/licenses/by-sa/4.0/

Article on data anonymization: https://www.science.org/doi/10.1126/sciadv.adn7053

Contact me:
My Mastodon: https://mstdn.social/@theeclecticdyslexic
My Matrix: https://matrix.to/#/@eclexic:matrix.org

Text based tutorial: https://github.com/Helium314/HeliBoard/wiki/Tutorial:-How-to-Contribute-Gesture-Data

It is recommended you install Heliboard through the F-Droid app, unless you know what you are doing!
How to install F-Droid: https://f-droid.org/en/docs/Get_F-Droid/
Heliboard on F-Droid: https://f-droid.org/en/packages/helium314.keyboard/
Heliboard on Github: https://github.com/Helium314/HeliBoard/releases

Gesture typing library links from MindTheGApps:
ARM64: https://gitlab.com/MindTheGapps/vendor_gapps/-/blob/fe250848941171fe339ca9a44bc9a42aefb0be7d/arm64/proprietary/product/lib64/libjni_latinimegoogle.so
ARM: https://gitlab.com/MindTheGapps/vendor_gapps/-/blob/fe250848941171fe339ca9a44bc9a42aefb0be7d/arm/proprietary/product/lib/libjni_latinimegoogle.so
X86_64: https://gitlab.com/MindTheGapps/vendor_gapps/-/blob/fe250848941171fe339ca9a44bc9a42aefb0be7d/x86_64/proprietary/product/lib64/libjni_latinimegoogle.so


Other ways to contribute:
Providing packaging scripts for Swipe-o-Scope:

  • for Windows
  • for macOS
  • for Flatpak. This is a big task; Swipe-o-Scope uses QT modules that are not currently supported by the KDE SDK (QTGraphs module). Swipe-o-Scope is also written using the PySide6 python library, rather than in C++. To build Swipe-o-Scope for flatpak, you are probably going to have to talk with KDE developers and the PySide6 baseapp maintainer. You will need them to update the SDK and the baseapp to support PyQTGraph. This is all in addition to needing to know a little about building flatpaks.
  • for Linux via means other than Flatpak (e.g. the AUR)

Providing input or code if you are knowledgeable about any of the following:

  • gesture typing using hand-designed algorithms... (bonus points if you have worked on a paper or product that you could help us make an open implementation of WITHOUT violating anyone's intellectual property)
  • gesture typing using neural nets and constrained compute, such as on mobile devices without TPUs... (unfortunately you may not be able to contribute here effectively until we have the data collected, organised, and released at the end of the collection period)
  • the JNI in Android... (bonus points if you have a working knowledge of the AOSP Latin IME JNI library)
  • natural language processing for next word prediction, specifically comparing the suitability of a set of candidate words against one another... (either by ngrams or any other low-compute method available to a mobile device with no internet connection)
  • building a diverse small-to-medium sized multi-lingual corpus of natural language text we could legally use to simulate context... (not stealing copyrighted content in bulk, like certain companies.)
  • making desktop apps more user-friendly... (Swipe-o-Scope doesn't currently give user feedback that would be helpful to anyone that doesn't feel comfortable in a terminal)
  • being patient while performing thorough code review and audits of rust code... (the gesture recognition library is most likely to be in rust, despite me being more experienced in other languages, as I am making a bet on it still being popular in a decade or three)

Full disclosure: out of concern for copyright issues and code quality, we will not be accepting ANY LLM generated contributions to this project. Neither in the form of code, nor corpus text. Thank you!

If you work on another on-screen keyboard, have the ability to collect data from it, and you want to add to the data set - you can contact me about some more particulars of the file format. Not all requirements of the file format are obvious and should not be assumed!

We are still in early stages, and this project is likely to continue for quite a while. It may continue well past the end date of me getting paid by the NLnet. Don't hesitate to reach out if you think you can help in some other way! We can use all the help we can get; gesture typing is a hard problem with a very high ceiling. Every little improvement matters!

Remember, even sharing this project around will be helpful at the moment. I can't be boosting this all the places I maybe should be; I have to be working on code, and this video took long enough!

I thought it might get some reach when cross-posted here

all 13 comments

sorted by: hot top controversial new old
[–] 15 points 8 months ago (6 children)

I like Heliboard except the autocorrect sucks balls. When I tried gboard - despite my reluctance - it was a pleasure. I really like Heliboard a lot but it’s infuriatingly bad when it comes to sensing what I meant to type.

  • source
  • hideshow 6 child comments
  • [–] 1 point 7 months ago (1 child)

    It's autocorrect works great for me. I never used gboard because google saw enough as it was. But it's a damn sight better than the Microsoft swyft which I used for awhile. I was constantly fighting the autocorrect on that.

    With heliboard I had some initial issues, but it turned out that it was system level autocorrect fucking with me. Once I disabled that and just used heliboard's it was (still is) excellent.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 10 points 8 months ago

    Did about a 101 words in Norwegian.

    GoddmDAMN "riksantikvarens" was hard. Skipped it.

  • source
  • [–] 4 points 8 months ago

    Cool stuff, thx for sharing!

  • source
  • [–] 3 points 8 months ago

    Fucking awesome, I'll be sure to contribute as many words as I can!

    Also, I remember FUTO Keyboard also had a little web browser based game to collect swipe data at some point. Wonder if they'd be willing to share? 👀

  • source
  • [–] 3 points 8 months ago (1 child)

    KDE is working on a unified on screen keyboard across all their desktop environments. Gnome has apsirstions for something similar for gtk. Intrgrating into these efforts?

  • source
  • hideshow 1 child comment
  • [–] 2 points 7 months ago

    Is florisboard's (also on fdroid) gesture typing open? It's not great, but it works...somewhat.

  • source
  • 36 comments in c/opensource@programming.dev view that post
    [–] 50 points 8 months ago (1 child)

    That's really cool. There are a lot of FOSS that need data collection to be accurate and useful. I wish more of them accept donation of anon data. e.g. I'd love to donate my location data in exchange of accurate transit data.

  • source
  • hideshow 1 child comment
  • [–] 26 points 8 months ago (6 children)

    Heliboard is awesome, and once you get it installed its swipe works well. I'm off Android now, but I'd turn on telemetry for Heliboard.

  • source
  • hideshow 6 child comments
  • [–] 19 points 8 months ago (3 children)

    Swipe in my experience does need work. I'm glad they're doing this.

    GBoard, before I switched, was like 95% accurate. I could trust it for blind typing. Heliboard with the google library is maybe 70% accurate and frustrating. "typing" in that last sentence was first "topping" the second "ripping" before I tapped it out.

    To this day nobody is as good as Swype was. RIP.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 9 points 8 months ago*

    Same experience here. Swype was the original and somehow after all these years still the best?! Or maybe I'm misremembering because my standards were lower back then.

    GBoard is noticeably better then Heliboard. I still use Heliboard but it is frustrating sometimes, to the point where I wonder if I should just start typing with my thumbs instead. Might be faster on the whole, since I sometimes lose 5-10 seconds correcting the swipos.

    I will start submitting gesture data ASAP. The keyboard world needs this.

  • source
  • parent
  • [–] 3 points 8 months ago (1 child)

    Swype was great but Swiftkey's (pre-Microsoft) word prediction made it my favorite. It was also able to pick the correct word without me having to care about swiping over the right letters. It seemed to work just by the shape I was tracing.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 9 points 8 months ago

    I've gotten used to Unexpected Keyboard for use with Termux, but I do agree the average folk need a keyboard for the people.

  • source
  • [–] 9 points 8 months ago

    I love this idea so I just enabled the swipe library we have now (closed, boooo) and submitted my first set of test data. It worked just like in the video here. Neat!

    If anyone sees this post and wants some more background on the keyboard and why it's so handy, here you go:
    https://www.howtogeek.com/i-tested-this-open-source-keyboard-for-a-month-and-it-replaced-gboard/

    NB: I am no shill just a happy user.

  • source
  • [–] 16 points 8 months ago (1 child)

    I haven't watched the video but Futo collected a swipe dataset which is MIT licensed.

    https://huggingface.co/datasets/futo-org/swipe.futo.org

  • source
  • hideshow 1 child comment
  • [–] 4 points 8 months ago (1 child)
  • [–] 3 points 8 months ago

    This has lit a fire under me, I'm gonna gesture so many words! Sucks that I have to download the proprietary library—I entirely stopped gesture typing because of that—but this is for the greater good. I'll exclusively use it for this data collection, and nothing else. Hopefully the word spreads, and we get a massive data set which makes a great FOSS gesture library on mobile possible.

  • source
  • [–] 2 points 8 months ago

    Being asked to gesture words I don't know, or Nierenfunktionsstörung and other long German words like that 😅☠️

  • source
  • [–] 1 point 8 months ago (16 children)

    FUTO Keyboard might be willing to share data. Its another open Open source keyboard. License purchase optional but encouraged.

  • source
  • hideshow 16 child comments
  • [–] 23 points 8 months ago (15 children)
  • [–] 19 points 8 months ago* (last edited 8 months ago) (14 children)

    https://github.com/futo-org/android-keyboard/

    Uses their own license which forbids commercial use but the source is available and you're free to modify and distribute it so...

    Not proper FOSS and doesn't meet the OSI definitions but colloquially speaking it looks pretty open to me.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 23 points 8 months ago (3 children)

    All this FUTO hate must be coming from big-FOSS so they can sell more free.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 3 points 8 months ago

    It's always inspiring to see random people on Lemmy calling out big-FOSS and sticking up for the little guy.

    (Mandatory /s in case people don't know that FUTO's founder is literally a tech billionaire)

  • source
  • parent
  • [–] 8 points 8 months ago (1 child)

    Yeah the more damning thing is the fascism. Anything from or in any way connected to Curtis Yarvin should get a wide berth. Rossman hooking his wagon to Yarvin is disappointing. But well within the norms for him sadly. (Kiwifarms association etc) I want to like him and support him for his right to repair work. But personally I can't.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 8 months ago* (7 children)

    If you modify it but then no one can use your modifications while at their work, then it's not much use being able to see the code.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 1 point 8 months ago* (last edited 8 months ago) (6 children)

    That's not how that works...

    It just means you can't modify it and make a profit from selling it or monetizing your modifications.

    (... Or try to sell it without modification, for that matter)

  • source
  • parent
  • hideshow 6 child comments
  • [–] 2 points 8 months ago* (5 children)

    Show me where the license supports your interpretation.

    You may modify the software only for non-commercial purposes such as personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, all without any anticipated commercial application.

    You may distribute the software or any part of its source code only if you do so free of charge for non-commercial purposes.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 2 points 8 months ago* (4 children)

    The parts you quoted, for one...

    "Non-commercial" is well defined.

    For a general overview: https://www.copyrighted.com/blog/copyright-commercial-use#what-is-commercial-use-in-copyright

  • source
  • parent
  • hideshow 4 child comments
  • [–] 2 points 8 months ago (3 children)

    Commercial use in copyright refers to the use of creative works, such as text, images, music, or videos, for financial gain.

    Just as I said, can't use the keyboard with modifications to write a work email. Since the license doesn't grant that permission, it's unsafe to assume that it does.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 8 months ago* (last edited 8 months ago) (2 children)

    When I was 14 I purchased a CD with a copy of the Audacity installer on it from a shady site for $30...

    I seriously doubt the authors intend you to uninstall your keyboard every time you need to write a work email; it's to stop you from profiting off of their work, just as you can't remove their branding and funding links when you re-distribute.

    I have CC BY-NC images as my desktop wallpaper on my work laptop... Now if my work was selling the laptop then it could become a problem, especially if the pretty art was part of the value proposition. While it merely existing might be a gray area under some hyper-pedantic interpretation, it's functionally irrelevant in the real world until there's money involved and the licensor can show damages.

    (Psst I also changed some behaviour in my GPL editor for personal use once without sharing my changes and also I also used that version a few times while writing proprietary commercial code... Pls don't tell Stallman)

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 8 months ago

    Psst I also changed some behaviour in my GPL editor for personal use once without sharing my changes and also I also used that version a few times while writing proprietary commercial code… Pls don’t tell Stallman

    As long as you didn't ship the edited binary of the GPL editor with your proprietary software, you are safe here.

  • source
  • parent
  • [–] 2 points 8 months ago*

    Personal anecdotes of successful piracy don't make for an open-source license agreement, otherwise I would be a minor king of open-source.

    If a database server's license says “only allowed to use for personal purposes”, it's rather obvious that I can't install it at my work. If a keyboard's license says “you may modify the software only for non-commercial purposes”, it's less clear, but also not apparent why the same interpretation shouldn't apply. Most importantly, copyright law doesn't allow willy-nilly use by default in the case of doubt.

    I also changed some behaviour in my GPL editor for personal use once without sharing my changes and also I used that version a few times while writing proprietary commercial code

    As the other commenter correctly pointed out, GPL only requires you to share your code if you distribute the compiled binary. And, being a fully FOSS license, GPL doesn't restrict commercial use of programs.

  • source
  • parent
  • [–] 0 points 8 months ago

    I like HeliBoard but the lack of a private swipe capability made it so clunky to use. I've been using FUTO Keyboard on Android which has the swipe feature but runs privately offline( no network permissions). The swipe accuracy isn't perfect but its better than typing each word.

  • source